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ABSTRACT 


The  1986  President’s  Blue  Ribbon  Commission  on  Defense  Management  (Packard 
Commission)  recommended  a  consolidation  of  the  Services’  materiel  acquisition 
regulations  under  an  umbrella  regulation,  DoD  5000.  Adopted  in  1991,  this  regulation 
created  four  levels  of  management  oversight,  Acquisition  Categories  (ACAT)  I  -  IV,  and 
it  encouraged  streamlining  or  tailoring  of  the  process  for  the  smaller  ACAT  III  and  IV 
programs. 

This  study  examines  the  streamlining  of  the  DoD  5000  procedures  across  the 
smallest  U.S.  Army  programs  (ACAT  IV).  It  investigates  differences  among  Commands 
and  which  streamlining  techniques  are  most  prevalent  and  useful.  The  basis  of  the  study 
consisted  of  literature  searches,  personal  and  telephonic  interviews,  and  a  questionnaire, 
with  questions  that  required  numeric  and  essay  responses,  that  was  distributed  Army- wide 
to  all  ACAT  IV  managers,  and  synthesis  of  this  collection  of  information.  A  total  of  19 
managers  responded. 

The  study  concluded  that  there  are  no  significant  differences  among  the  U.S.  Army 
Commands  concerning  the  application  of  acquisition  streamlining  techniques.  In  addition, 
it  identified  which  techniques  were  most  prevalent,  which  saved  the  most  program  time 
and  funding,  and  the  reasons  for  differences  among  Commands  and  between  small  and 
large  programs.  The  study  recommends  that  the  Army  establish  an  ACAT  IV  baseline 
development  management  process  that  incorporates  the  most  prevalent  and  useful 
streamlining  techniques. 
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EXECUTIVE  SUMMARY 


In  the  late  1980’s,  the  President’s  Blue  Ribbon  Commission  on  Defense 
Management  (Packard  Commission)  in  A  Quest  for  Excellence,  Final  Report  to  the 
President  recommended,  and  the  Department  of  Defense  (DoD)  Defense  Management 
Review  instituted,  a  consolidation  of  the  Services’  materiel  acquisition  regulations  under 
an  umbrella  regulation,  DoD  5000.  This  regulation  was  adopted  in  1991  and  it  changed 
many  aspects  of  how  the  Services  manage  and  conduct  material  development  programs. 
The  system  adopted  by  DoD  created  four  levels  of  management  oversight,  Acquisition 
Categories  (AC AT)  I  -  IV. 

The  new  DoD  5000  series  concentrates  on  the  management  oversight  required 
for  large  ACAT  I  programs.  The  regulations  outline  the  processes  required  of  AC  AT  I 
managers,  and  encourages  tailoring,  or  streamlining  of  the  process  by  ACAT  II,  III,  and 
IV  managers,  while  not  allowing  for  across-the-board  Service-level  policies. 

The  purpose  of  this  research  was  to  analyze  the  streamlining  of  DoD  5000 
procedures  and  processes  across  all  of  the  smallest  U.S.  Army  programs  (ACAT  IV); 
to  investigate  differences  among  various  U.S.  Army  Commands  and  determine  which 
streamlining  techniques  were  most  prevalent,  useful,  and  the  circumstances  that  cause 
them  to  be  successful.  Individual  managers  need  to  streamline  to  effectively  manage 
program  risk,  but  they  lack  key  data  necessary  for  conducting  proper  risk  assessments. 
This  study  sought  to  provide  this  missing  risk  assessment  data.  Specific  purposes  were 
to  determine  whether  Commands  need  assistance  in  gaining  knowledge  concerning  all  of 
the  streamlining  techniques  available  (being  used  by  others)  and  to  provide  the  ACAT 
IV  managers  with  tools  to  assist  them  in  their  streamlining  decision-making. 

Primary  Research  Question:  What  acquisition  streamlining  techniques  are  utilized  by 
U.S.  Army  ACAT  IV  managers? 


!X 


Subsidiary  Research  Questions: 

(1)  How  do  U.S.  Army  Commands  streamline  ACAT IV  programs  differently?  - 
Comparing  the  techniques  utilized  Command-by-Command  . 

(2)  How  do  the  various  streamlining  techniques  compare  with  respect  to  program 

cost  savings?  -  Comparing  technique-to-technique. 

(3)  How  do  the  various  streamlining  techniques  compare  with  respect  to  program 

schedule  reduction?  -  Comparing  technique-to-technique. 

The  methodology  utilized  for  this  effort  was  a  management  problem  analysis  using 
both  quantitative  and  qualitative  techniques.  The  qualitative  techniques  involved  the  use 
of  interviews  to  determine  the  opinions  of  the  AMC  acquisition  policy  Subject  Matter 
Experts  concerning  the  validity  and  breadth  of  the  techniques  to  be  investigated.  The 
information  from  this  process  was  assimilated  into  a  broad-based  questionnaire  that  was 
distributed  Army-wide  to  all  ACAT  IV  development  managers,  with  questions  that 
required  numeric  and  essay  responses.  The  numeric  responses  were  quantitatively 
analyzed,  and  the  essay  responses  were  used  to  find  elaboration/causal  relationships  for 
specific  responses  and/or  trends  observed  in  the  data. 

In  support  of  this  thesis,  39  questionnaires  were  distributed  to  the  ACAT  IV 
development  managers  at  various  U.S.  Army  AMC  Commands.  Two  Commands, 
CECOM  and  MICOM,  were  eliminated  from  the  survey  as  is  was  determined  that  all  of 
their  development  programs  were  at  the  ACAT  III  level  or  higher.  A  total  of  19 
responses  were  obtained,  with  some  Commands  providing  limited  responses.  The  data 
that  was  received  was  deemed  to  be  valid,  and  the  analysis  was  conducted  on  a 
predominately  qualitative  basis. 

From  the  study,  it  was  concluded  that  cross-fertilization  of  streamlining  concepts 
exists  among  the  U.S.  Army  Commands.  The  information  is  available  for  all  Commands 
to  use.  In  addition,  relative  usage  levels  for  the  various  techniques  is  consistent  among 
Natick  and  the  other  U.S.  Army  Commands.  For  those  techniques  where  the  usage 
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differed  the  most,  the  reasons  were  disagreement  with  the  value  or  risk  of  the  technique, 
or  specific  commodity/Command  differences. 

A  number  of  streamlining  techniques  are  used  routinely  by  AC  AT  IV  programs. 
The  risk  levels  associated  with  the  combining  of  Milestones  and  test  phases  are 
considered  acceptable  for  programs  at  this  lower  level  of  complexity.  Although 
quantifiable  and  useful  ranges  for  the  levels  of  anticipated  savings  for  the  techniques 
could  not  be  projected  from  the  research,  the  following  can  be  concluded  as  the  key 
techniques  utilized  by  most  AC  AT  IV  managers. 

1.  Combining  of  Milestones  0  and  I,  and/or  I  and  II. 

2.  Combining  of  test  phases,  both  in  Dem/Val  and  EMD,  for  Natick  managers. 

3.  Concurrency  of  test  phases,  both  in  Dem/Val  and  EMD,  for  non-Natick 

managers. 

4.  Combining  development  deficiency  correction  validation  testing  with  First 

Article  Testing. 

Small  and  large  ACAT  IV  program  managers  are  in  close  agreement  concerning 
the  relative  cost  savings  associated  with  each  of  the  streamlining  techniques  in  the  study. 
The  two  areas  where  large  programs  deviated  from  the  trend  are:  (1)  combining  EMD 
DT  and  IOT&E,  and  (2)  using  a  single  contract  for  Dem/Val  and  EMD,  both  of  which 
may  be  due  to  the  increased  complexity  of  larger  programs. 

It  is  recommended  that  the  results  of  this  study  be  broadly  distributed  to  ACAT 
IV  managers.  Even  though  the  results  provide  no  quantitative  assistance  for  ACAT  IV 
managers  concerning  expected  savings  levels  for  the  various  streamlining  techniques 
investigated,  the  study  does  provide  valuable  information  that  should  assist  managers  in 
their  streamlining  decision-making. 

It  is  further  recommended  that  the  U.S.  Army  revise  the  baseline  development 
process  for  ACAT  IV  programs,  to  one  that  incorporates  the  above  key  streamlining 
techniques  as  standard  procedures  from  which  requirements  are  either  increased,  or 
decreased.  In  place  of  the  current  system,  where  the  manager  must  begin  with  the  full 
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AC  AT  I/II  process  and  struggle  with  the  acquisition  team  members  to  cut  out  elements, 
this  recommendation  would  put  acquisition  team  members  in  a  position  of  justifying  the 
need  for  adding  requirements  versus  the  manager  having  to  justifying  their  exclusion. 


I.  INTRODUCTION 


In  the  late  1980’s,  a  Department  of  Defense  (DoD)  review  of  acquisition 
management  practices  resulted  in  the  consolidation  of  the  Services’  materiel  acquisition 
regulations  under  an  umbrella  regulation,  DoD  5000.  This  regulation  was  adopted  in 
1991  and  it  changed  many  aspects  of  how  the  Services  manage  and  conduct  material 
development  programs.  The  system  adopted  by  DoD  created  four  levels  of  management 
oversight,  Acquisition  Categories  (ACAT)  I  -  IV.  Application  of  these  categories  is  based 
on  program  size  and/or  the  special  interests  of  the  Office  of  the  Secretary  of  Defense  and 
the  Service’  headquarters.  The  largest  and  most  significant  programs  are  ACAT  I  and, 
conversely,  the  smallest  and  least  significant  are  ACAT  IV.  [Ref.  1] 

One  aspect  of  the  new  DoD  5000  series  that  did  not  exist  in  the  old  regulations 
is  that  the  Services  are  now  prohibited  from  regulatory  supplementation.  This  was 
instituted  to  specifically  eliminate  the  possibility  of  the  Services  reverting  to  Service- 
specific  policy.  Unfortunately,  it  also  creates  an  inability  for  a  Service  to  implement  an 
across-the-board  policy  for  streamlining  the  procedures  and  processes  outlined  in  DoD 
5000  without  OSD  approval.  DoD  5000  does,  however,  encourage  streamlining  or 
tailoring  the  process  on  a  project-by-project  basis.  [Ref.  2] 

A.  PROBLEM  DEFINITION 

The  purpose  of  this  research  is  to  analyze  the  streamlining  of  DoD  5000 
procedures  and  processes  across  all  U.S.  Army  ACAT  IV  programs  as  well  as  to 
investigate  differences  among  Commands  and  determine  which  streamlining  techniques 
are  most  prevalent,  useful,  and  the  circumstances  that  cause  them  to  be  successful.  From 
the  researcher’s  twelve  years  of  experience  in  ACAT  IV  development  management  and 
discussions  with  colleagues  within  the  U.S.  Army,  streamlining  of  ACAT  IV  programs 
has  been  taking  place,  but  on  an  ad-hoc,  trial-and-error  basis.  Comparison,  or  cross- 
fertilization,  of  information  among  individual  managers  and  Commands  (concerning 
which  streamlining  techniques  have,  or  have  not  worked)  has  only  taken  place  through 
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informal  channels.  It  can  be  expected  that  within  each  Command,  the  information  has 
been  disseminated  fairly  well,  since  each  Command  has  a  support  organization  on 
acquisition  policy  that  is  involved  in  each  project’s  planning.  [Ref.  3]  However,  the  ability 
to  pass  ideas  between  Commands  may  be  incomplete,  since  there  is  not  a  formal  means 
to  channel  this  information.  Therefore,  this  study  investigates  the  various  streamlining 
techniques  utilized  by  ACAT  IV  managers  among  the  various  Commands  and  seek  to 
define  the  causal  relationships  for  their  use.  Specific  purposes  are  to  determine  whether 
Commands  need  assistance  in  gaining  knowledge  concerning  the  streamlining  techniques 
available  and  to  provide  the  ACAT  IV  managers  with  tools  to  assist  them  in  their 
streamlining  decision-making.  This  study  draws  conclusions  regarding  which  techniques 
are  most  useful,  which  need  to  be  advertised  to  other  Commands,  and  the  key  factors 
that  influence  the  streamlining  decision-making  process. 

1.  Research  Questions 

Primary  Research  Question:  What  acquisition  streamlining  techniques  are 
utilized  by  U.S.  Army  ACAT  IV  managers? 

Subsidiary  Research  Questions: 

(1)  How  do  U.S.  Army  Commands  streamline  ACAT  IV  programs  differently? 
(Comparing  the  techniques  utilized  Command-by-Command). 

(2)  How  do  the  various  streamlining  techniques  compare  with  respect  to  program 
cost  savings?  (Comparing  technique-to-technique). 

(3)  How  do  the  various  streamlining  techniques  compare  with  respect  to  program 
schedule  reduction?  (Comparing  technique-to-technique). 

2.  Expected  Benefit 

This  research  is  intended  to  determine  if  tools  exist  that  ACAT  IV  managers  can 
use  to  improve  their  acquisition  streamlining  decisions,  and,  if  so,  provide  these 
managers  with  alternative  streamlining  techniques  not  previously  considered  by  their 
Commands.  The  results  are  also  intended  to  assist  policy  makers  and  support  commands 
when  they  consider  the  adoption  of  streamlining  policies  for  ACAT  IV  programs,  or  their 
application  to  new  program  starts. 
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This  research  effort  investigates  one  aspect  of  the  management  of  small  programs, 
the  streamlining  techniques  that  the  experts,  the  current  AC  AT  IV  managers,  have 
deemed  beneficial  to  their  programs.  This  research  is  unique  in  that  it  focuses  on  the 
smallest  acquisition  programs.  As  noted  by  the  lack  of  related  research  discovered  in  the 
literature  search  for  this  research  project  (see  the  literature  search  results  below),  efforts 
at  acquisition  reform  and  streamlining  have  been  almost  exclusively  concentrated  on  the 
high-visibility,  high-cost  weapon  system  acquisitions  (the  AC  AT  I  and  IIs).  Thus,  the 
smaller  programs  have  not  been  considered  in  depth,  and  they  have  been  required  to  use 
the  DoD  5000  policies  and  procedures  tailored  for  the  larger  programs.  This  effort  is 
sponsored  by  the  U.S.  Army  Natick  Research,  Development,  and  Engineering  Center, 
the  organization  containing  the  majority  of  the  U.S.  Army’s  AC  AT  IV  development 
programs. 

3.  Boundaries 

This  study  investigates  acquisition  streamlining  techniques  within  the  context  of 
tailoring  acquisition  procedures  and  documentation,  per  DoD  5000  direction,  for  the 
development  of  new  U.S.  Army  ACAT  IV  systems.  The  research  includes  analysis  of 
all  available  on-going  and  recently  completed  U.S.  Army  ACAT  IV  development 
programs,  except  as  noted  in  the  limitations  and  constraints  below.  This  is  not  an 
evaluation  of  the  use  of  commercial  versus  military  specifications  and  standards,  and  it 
excludes  non-developmental  and  classified  ACAT  IV  programs,  as  well  as  non-regulatory 
program  acceleration  techniques  such  as  improved  business  management  processes,  Total 
Quality  Management,  program  management  software  tools,  etc. 

4.  Limitations  and  Constraints 

Programs  that  are  in  production  are  excluded,  because  of  the  difficulty  in 
accessing  past  managers  (who  have  been  reassigned)  and  the  time  and  funding  constraints 
of  this  research  effort.  However,  this  should  not  be  a  significant  limitation  since  the 
ACAT  system  has  only  existed  for  three  years,  and  these  programs  largely  would  have 
been  developed  under  the  old  system.  Also,  the  amount  of  funding  and  time  available  did 
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not  allow  for  travel  to  conduct  face-to-face  interviews  with  each  of  the  ACAT  IV 
managers. 

As  will  be  discussed  later  in  this  paper,  the  main  support  instrument  for  this 
research,  a  written  questionnaire,  had  a  low  response  rate.  Investigation  of  the  lack  of 
response  found  that:  there  are  not  as  many  ACAT  IV  development  managers  as 
anticipated,  some  managers  are  not  as  knowledgeable  as  assumed,  and  some  managers 
did  not  have  time  available  to  complete  the  questionnaire.  Because  of  this  limitation,  the 
analysis  is  predominately  qualitative,  instead  of  quantitative,  in  nature. 

5.  Assumptions 

It  is  assumed,  from  the  researcher’s  experience  at  the  U.S.  Army  Natick 
Research,  Development,  and  Engineering  Center,  that  streamlining  is  a  concern  of  all 
U.S.  Army  ACAT  IV  managers.  It  is  also  assumed  that  the  ACAT  IV  managers  are  the 
most  knowledgeable  as  to  which  streamlining  techniques  are  most  effective,  because  it 
is  their  mission  to  manage  these  decisions,  and  they  should  have  an  established 
knowledge  base  from  their  experiences  and  discussions  with  technical  support  staffs. 

Additional  assumptions  are  as  follows:  generalized  relationships  exist  among 
ACAT  IV  programs  (with  respect  to  streamlining  initiatives)  that  can  be  analyzed  within 
the  constraints  of  this  study;  as  all  ACAT  IV  programs  follow  the  same  general  process 
(and  are  relatively  low  risk)  qualitative  comparisons  of  ACAT  IV  processes  and 
streamlining  techniques  should  be  possible;  managers  aren’t  able  to  exactly  quantify  the 
impacts  of  individual  streamlining  initiatives;  and  a  sufficient  quantity  of  ACAT  IV 
managers  exist,  and  a  sufficient  number  of  the  managers  are  willing  to  participate,  to 
provide  a  meaningful  basis  for  analysis. 

B.  METHODOLOGY 

The  approach  utilized  for  this  effort  was  a  management  problem  analysis  through 
a  survey  of  all  U.S.  Army  ACAT  IV  project  managers/officers  for  their  expert  opinions 
concerning  streamlining  techniques.  They  were  asked  if  they  have,  or  would  utilize 
specific  streamlining  initiatives,  and  rate/comment  on  the  cost  and  schedule  savings 
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associated  with  each.  Also,  comments  were  requested  concerning  the  key  factors  that  led 
to  the  above  ratings.  Results  were  then  evaluated  comparatively,  both  by  Command  and 
by  technique. 

Because  the  population  of  AC  AT  IV  managers  is  geographically  dispersed,  the 
data  collection  methods  used  were  telephone  interviews  and  written  questionnaires.  The 
broad,  investigative  nature  of  the  questions  being  addressed  in  this  initial  study  of  AC  AT 
IV  management  made  this  methodology  acceptable.  In  addition,  the  effects  of  this 
constraint  were  mitigated  through  careful  construction  of  the  questionnaire  materials.  For 
instance,  the  telephone  interviews  established  a  mutually  exclusive  and  exhaustive  set  of 
streamlining  techniques  for  use  in  the  written  questionnaires  (allowing  for  closed-ended 
questions),  and  the  size  of  the  respondents  programs  were  established  (in  terms  of  cost 
and  duration)  to  ensure  that  the  comparative  analysis  encompassed  programs  of  similar 
size. 

C.  LITERATURE  REVIEW 

A  review  of  the  available  literature  revealed  no  directly  related  past  efforts.  This 
result  is  understandable,  given  the  general  focus  within  the  DoD  on  the  major  systems, 
and  the  newness  of  the  ACAT  designation  system. 

A  number  of  previous  studies  did  consider  acquisition  streamlining,  but  at  a  much 
broader  level.  One  of  these  reviews  was  the  President’s  Blue  Ribbon  Commission  of 
1986,  discussed  in  Chapter  II,  which  was  the  impetus  for  the  new  DoD  5000  series. 
These  studies  were,  however,  non-analytical,  expert  panel  reviews  of  the  overall  process. 
None  analyzed  streamlining  across  projects,  and  none  considered  non-major,  or  ACAT 
IV  programs. 

One  previous  study  was  closely  related  to  the  purpose  and  methodology  of  this 
research  effort.  A  1991  study  by  Mr.  Henry  Jehan,  for  the  U.S.  Army  War  College, 
examined  the  impact  of  the  Packard  Commission  on  the  U.S.  Army  ACAT  I,  II,  and  III 
managers  (the  population  of  U.S.  Army  development  managers  specifically  not  within 
the  scope  of  the  current  research  effort).  Mr.  Jehan  utilized  a  questionnaire  to  all 
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available  members  of  the  population  (the  Program  and  Product  Managers),  which 
corresponded  closely  to  the  methodology  planned  for  this  effort.  Mr.  Jehan’s  research 
questions  did  not,  however,  investigate  the  same  issues  as  the  current  research.  His 
research  question  was  stated  as  follows: 

The  central  question  this  study  attempts  to  answer  is:  how  well  has 
the  Army  implementation  embodied  the  recommendations  of  the  Packard 
Commission?  Specifically,  do  the  current  Army  management  structures 
conform  to  the  structures  recommended  by  the  Commission?  And  more 
importantly,  does  the  Army  acquisition  process  implement  the  spirit  of  the 
Commission’s  recommendations? [Ref.  4] 

Mr.  Jehan’s  report  provides  a  useful  general  outline  for  conducting  the  current 
study.  It  confirms  the  validity  of  the  methodology  and  it  contains  data  sources  and 
questionnaire  lessons  learned  that  can  be  applied  to  this  research. 

The  two  previous  acquisition  streamlining  theses  at  the  Naval  Postgraduate  School 
were  completed  prior  to  the  advent  of  the  ACAT  system.  Those  studies  focused  on  the 
investigation  of  streamlining  within  a  specific  program/Command  and  they  consisted 
mainly  of  historical  assessments  of  that  organization’s  streamlining  actions.  The  first, 
written  by  Mary  Walsh  and  published  in  December  1986,  utilized  reviews  of  regulations 
and  program  documents,  as  well  as  interviews  and  resulted  in  a  case  study.  The  purpose 
was  to  analyze  the  effectiveness  of  the  then  new  Army  Streamlined  Acquisition  Program 
(ASAP),  the  Navy  T-45  Trainer  development  program,  and  the  Army  and  Navy 
implementation  of  the  since  superseded  DoDI  5000.43,  "Acquisition 
Streamlining". [Ref.  5]  The  second,  written  by  Michelle  McKeever  and  published  in  June 
1987 ,  reviewed  the  acquisition  streamlining  activities  within  the  Space  and  Naval  Warfare 
Systems  Command.  It  also  followed  a  case  study  methodology  using  interviews  and 
document  reviews.  [Ref.  6] 

From  this  literature  review,  and  the  researcher’s  experience  with  past  U.S.  Army 
management  studies  of  the  acquisition  process  that  all  concentrated  on  major  weapon 
systems,  it  is  felt  that  this  research  is  in  a  new  area  of  investigation. 
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D.  SUMMARY  OF  FINDINGS 


1.  Conclusions 

From  this  study,  it  can  be  concluded  that  cross-fertilization  of  streamlining 
concepts  exists  among  the  U.S.  Army  Commands.  The  information  is  available  for  all 
Commands  to  use.  In  addition,  relative  usage  levels  for  the  various  techniques  is 
consistent  among  Natick  and  the  other  U.S.  Army  Commands.  For  those  techniques 
where  the  usage  differed  the  most,  the  reasons  were  disagreement  with  the  value  or  risk 
of  the  technique,  or  specific  commodity /Command  differences. 

A  number  of  streamlining  techniques  are  used  routinely  by  AC  AT  IV  programs. 
The  risk  levels  associated  with  the  combining  of  Milestones  and  test  phases  are 
considered  acceptable  for  programs  at  this  lower  level  of  complexity.  Although 
quantifiable  and  useful  ranges  for  the  levels  of  anticipated  savings  for  the  techniques 
could  not  be  projected  from  the  research,  the  following  can  be  concluded  as  the  key 
techniques  utilized  by  most  AC  AT  IV  managers. 

1.  Combining  of  Milestones  0  and  I,  and/or  I  and  II. 

2.  Combining  of  test  phases,  both  in  Dem/Val  and  EMD,  for  Natick  managers. 

3.  Concurrency  of  test  phases,  both  in  Dem/Val  and  EMD,  for  non-Natick 

managers. 

4.  Combining  development  deficiency  correction  validation  testing  with  First 

Article  Testing. 

In  addition,  ACAT  IV  programs  generally  do  not  require  COEA  or  STAR 
program  documents,  and  the  acceleration  of  independent  evaluation  and  assessment 
reporting  for  Milestone  Decision  Reviews  is  commonly  done  to  accelerate  the  process 
during  the  time-sensitive  weeks  prior  to  a  major  decision  review. 

Small  and  large  ACAT  IV  program  managers  are  in  close  agreement  concerning 
the  relative  cost  savings  associated  with  each  of  the  streamlining  techniques  in  the  study. 
The  two  areas  where  large  programs  deviated  from  the  trend  are:  (1)  combining  EMD 
DT  and  IOT&E,  and  (2)  using  a  single  contract  for  Dem/Val  and  EMD,  both  of  which 
may  be  due  to  the  increased  complexity  of  larger  programs. 
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2.  Recommendations  Regarding  ACAT  IV  Management 

It  is  recommended  that  the  results  of  this  study  be  broadly  distributed  to  ACAT 
IV  managers.  Even  though  the  results  provide  no  quantitative  assistance  for  ACAT  IV 
managers  concerning  expected  savings  levels  for  the  various  streamlining  techniques 
investigated,  the  study  does  provide  valuable  information  that  should  assist  managers  in 
their  streamlining  decision-making. 

It  is  further  recommended  that  the  U.S.  Army  revise  the  baseline  development 
process  for  ACAT  IV  programs,  to  one  that  incorporates  the  above  key  streamlining 
techniques  of  combining  of  Milestones  I  and  II  concurrent  test  phases  in  EMD  as 
standard  procedures  from  which  requirements  are  either  increased,  or  decreased.  In 
addition,  it  is  recommended  that  the  alternative  process  exclude  the  requirements  for 
COEA  and  STAR,  except  on  an  exception  basis  only.  In  place  of  the  current  system, 
where  the  manager  must  begin  with  the  full  ACAT  I/II  process  and  struggle  with  the 
acquisition  team  members  to  cut  out  elements,  this  recommendation  would  put  acquisition 
team  members  in  a  position  of  justifying  the  need  for  adding  requirements  versus  the 
manager  having  to  justifying  their  exclusion. 

3.  Recommendations  for  Further  Research 

Considering  the  lack  of  statistically  significant  results  from  this  study,  it  is 
recommended  that  further  study  be  conducted  that  concentrates  on  the  most  significant 
techniques,  investigating  more  in-depth  the  establishment  of  useful  statistics,  such  as 
confidence  intervals,  for  the  expected  levels  of  cost  and  schedule  savings. 

Based  on  the  small  number  of  ACAT  IV  managers  in  the  U.S.  Army  it  is 
recommended  that  future  research  in  this  area  consider  both  ACAT  III  and  ACAT  IV 
managers  to  study  the  differences  between  the  two  groups,  and,  if  they  prove  to  be 
similar,  combine  the  two  for  statistical  information  concerning  expected  savings. 

Since  the  Marine  Corps  has  many  more  ACAT  III  and  IV  programs  than  ACAT 
I  and  II,  and  they  are  actively  streamlining  wherever  possible,  it  is  recommended  that 
further  research  be  conducted  that  considers  both  Army  and  Marine  Corps  programs. 
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H.  BACKGROUND 


A.  THE  DOD  5000  SERIES 

During  the  early  1980s,  major  problems  with  DoD  management  of  weapon 
systems  acquisitions  had  become  issues  of  concern  within  the  Executive  Branch  and 
Congress.  In  addition,  the  press  and  the  American  public  were  raising  concerns  of 
perceived  fraud,  waste,  and  abuse.  Cost  overruns  on  weapon  systems,  outrageous  spare 
parts  prices,  and  alleged  fraudulent  activities  within  the  Department  and  it’s  contractors 
brought  an  overwhelming  desire  for  reforms.  [Ref.  7]  In  response  to  these  problems,  by 
Executive  Order  12526,  President  Reagan  chartered  a  commission  in  July  1985  to 
conduct  a  study  of  Department  of  Defense  management  problems.  The  President’s  Blue 
Ribbon  Commission  on  Defense  Management,  more  popularly  known  as  the  "Packard 
Commission"  after  the  chairman  of  the  Commission,  former  Deputy  Secretary  of  Defense 
Mr.  David  Packard,  took  a  broad  and  in-depth  look  at  defense  weapon  system  acquisition 
issues  and  their  root  causes.  Their  charter  said  that: 

The  Commission  shall  study  the  issues  surrounding  defense 
management  and  organization....  The  primary  objective  of  the 
Commission  shall  be  to  study  defense  management  policies  and 
procedures,  including  the  budget  process,  the  procurement  system, 
legislative  oversight,  and  the  organizational  and  operational  arrangements, 
both  formal  and  informal,  among  the  Office  of  the  Secretary  of  Defense, 
the  Organization  of  the  Joint  Chiefs  of  Staff,  the  unified  and  Specified 
Command  system,  the  Military  Departments,  and  the  Congress.  In 
particular,  the  Commission  shall: 

1.  Review  the  adequacy  of  the  defense  acquisition  process, 
including  the  adequacy  of  the  defense  industrial  base,  current  law 
governing  Federal  and  Department  of  defense  procurement  activities, 
departmental  directives  and  management  procedures,  and  the  execution  of 
acquisition  responsibilities  within  the  Military  Departments.  (Emphasis 
added)  [Ref.  8] 

The  goals  of  the  Commission  included  strengthening  and  streamlining  the  control 
and  supervision  of  the  acquisition  process.  One  of  the  many  significant  recommendations 


9 


proposed  by  the  Commission  in  their  June  1986  report  was  the  consolidation  and 
simplification  of  acquisition  laws  and  regulations.  [Ref.  9]  A  follow-on  review  of  the 
regulations  governing  the  Service’s  acquisition  processes  led  to  a  July  1989  Defense 
Management  Review  (DMR)  Decision  to  consolidate  the  material  acquisition  regulations 
of  the  Services  under  an  umbrella  set  of  regulations.  The  basis  for  this  effort  was  to 
eliminate  unnecessary  Service-unique  oversight  and  bureaucracy,  and  streamline  the 
overall  process. [Ref.  10J 

The  new  DoD  Directive  5000.1,  "Defense  Acquisition";  DoD  Instruction  5000.2, 
"Defense  Acquisition  Management  Policies  and  Procedures";  and  DoD  Manual  5000.2- 
M,  "Defense  Acquisition  Management  Documents  and  Reports",  all  dated  23  February 
1991,  replaced  the  1987  versions  of  DoDD  5000.1  and  DoDI  5000.2.  Publication  of 
these  documents  was  the  result  of  a  two  year  project  under  three  Under  Secretaries  of 
Defense  for  Acquisition.  More  than  60  previous  directives,  instructions,  manuals  and 
memoranda  were  canceled  by  these  new  documents. 

This  new  "5000  Series"  provided  a  single  uniform  acquisition  system  for  all  of 
the  Services’  acquisition  programs:  major  defense  acquisition  programs,  non-major 
defense  acquisition  programs,  and  highly  sensitive  classified  programs.  The  DODI 
5000.2  states  that  the  acquisition  process  shall  be  structured  in  discrete  phases  separated 
by  major  milestones.  Under  the  old  DoDI  5000.2,  there  were  six  major  milestones  (O 
through  V),  and  five  phases.  The  new  DoDI  5000.2  provides  for  five  major  milestones 
with  phases  as  shown  in  Figure  l.[Ref.  11] 


Figure  1.  Acquisition  Milestones  and  Phases. 
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B.  THE  ACQUISITION  CATEGORY  SYSTEM 


Under  the  new  DoD  5000  series,  defense  acquisition  programs  are  divided  into 
acquisition  categories  (ACATs)  using  the  criteria  shown  in  Figure  2.  The  AC  AT 
designations  indicate  which  programs  come  under  the  Defense  Acquisition  Board  (DAB) 
or  under  a  DoD  Component  Acquisition  Executive  for  review  and  oversight.  The  DAB 
is  the  senior  DoD  acquisition  review  board  and  it  is  chaired  by  the  Under  Secretary  of 
Defense  for  Acquisition.  The  Vice  Chairman  of  the  Joint  Chiefs  of  Staff  is  the  Vice- 
Chair.  Other  members  of  the  Board  are:  the  Deputy  Under  Secretary  of  Defense  for 
Acquisition;  Service  Acquisition  Executives  of  the  Army,  Navy,  and  Air  Force;  the 
Director  of  Defense  Research  and  Engineering;  the  Assistant  Secretary  of  Defense  for 
Program  Analysis  and  Evaluation;  the  Comptroller  of  the  Department  of  Defense;  the 
Director  of  Operational  Test  and  Evaluation;  the  appropriate  DAB  Committee  Chair;  and 
the  DAB  Executive  Secretary.  The  DoD  Component  Acquisition  Executive  is  a  single 
official  within  each  DoD  Component  who  is  responsible  for  all  acquisition  functions 
within  that  Component.  For  the  Army,  this  individual  is  the  Assistant  Secretary  of  the 
Army  for  Research,  Development,  and  Acquisition.  [Ref.  12] 

The  programs  with  the  lowest  cost  and  risk,  those  that  were  classified  as  non¬ 
major  under  the  old  system,  are  classified  as  ACAT  III  and  IV  under  the  new  series.  A 
major  distinction  between  ACAT  III  and  IV  programs  and  ACAT  I  and  II  programs  is 
that  the  DoD  Component  Acquisition  Executives  may  delegate  decision  authority  for 
ACAT  III  and  IV  programs  to  the  lowest  level  deemed  appropriate  within  their  respective 
organizations.  [Ref.  13]  For  Army  ACAT  IV  systems,  this  Milestone  Decision  Authority 
(MDA)  designation  is  generally  established  at  the  Major  Subordinate  Command  (MSC) 
level  within  the  Army  Material  Command  (AMC).  See  Figure  3  for  a  schematic  of  the 
applicable  U.S.  Army  Command  structure. [Ref.  14] 

Within  U.S.  Army  practices,  any  program  that  does  not  meet  the  criteria  for 
ACAT  I  or  II,  but  has  a  Program  or  Product  Manager  assigned,  receives  an  ACAT  III 
designation.  All  other  programs  are  designated  as  ACAT  IV  by  default,  and  are  managed 
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ACAT 


SELECTION  CRITERIA 


DESIGNATION  AUTHORITY 


MILESTONE  DECISION 
AUTHORITY 


1 

•  A  program  not  classified  as 
rtigniy  sensitive  by  the  Secretary 
oi  Defense  that  has: 

**  Seen  designated  by  the 

Under  Secretary  of  Defense 
(Acquisition)  as  an 
acquisition  category  i 
program  or  is 

Estimated  by  the  Under 
Secretary  to  require: 

-  An  eventual  expenditure 
for  research,  development, 
test,  and  evaluation  of  more 
than  $200  million  in  fiscal 
year  1980  constant  dollars 
(approximately  S300  million 
in  fiscal  year  1990  constant 
dollars);  or 

•  An  eventual  expenditure 
for  procurement  of  more 
than  SI  billion  in  fiscal  year 
1980  constant  dollars 
(approximately  $1.8  billion 
in  fiscal  year  1990  constant 
dollars) 

•  Under  Secretary  of  Oefense 
(Acquisition) 

•  Acquisition  category  1  programs 
are  further  designated  by  the 
Under  Secretary  of  Defense 
(Acquisition)  as  either  requiring 
decision  by  the: 

••  Under  Secretary  -  ACAT  1  D 

**  Component  Head  -  ACAT  I C 

•  aCATI  0  -  Under  Secretary  of 
Qefeme  (Acquisition) 

•  ACAT  1 C  -  OoO  Component  Head 
or,  if  delegated,  the  OoO 
Component  Acquisition 

Executive 

II 

•  A  program  not  meeting  the 
criteria  for  category  1  that  has: 

••  Seen  designated  by  the  OoO 
Component  Head  as  an 
acquisition  category  II 
program  oris 

Estimated  by  the  OoO 
Component  Head  to  require: 

-  An  eventual  exoenditure 
for  research,  development, 
test,  and  evaluation  of  more 
than  $75  million  in  fiscal 
year  1980  constant  dollars 
(approximately  SI  IS  million 
in  fiscal  year  1990  constant 
dollars);  or 

•  An  eventual  expenditure 
for  procurement  of  more 
than  $300  million  in  fiscal 
year  1980  constant  dollars 
(approximately  S540  million 
in  fiscal  year  1990  constant 
dollars) 

•  OoO  Component  Head  or.  if 
delegated,  the  OoO  Component 
Acquisition  Executive 

•  OoO  Component  Head  or,  i  f 
delegated,  the  OoO  Component 
Acquisition  Executive 

III 

*  Programs  not  meeting  the 
cntena  for  category  1  and  11  that 
have  been  designated  category 

III  by  the  OoO  Component 
Acquisition  Executive 

*  OoO  Component  Acquisition 
Executive 

•  Lowest  level  deemed  appropriate 
by  the  designation  authority 

IV 

•  All  other  acquisition  programs 
for  which  the  milestone  decision 
authority  should  be  delegated  to 
a  level  below  that  required  for 
category  HI 

•  OoO  Component  Acquisition 
Executive 

«  Lowest  level  deemed  appropriate 
by  the  designation  authonty 

Figure  2.  The  Acquisition  Categories  and  Milestone  Decision  Authority,  from  Ref.  2 


12 


U.S.  Army  Organizational  Structure 

ACAT  IV  Programs 
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Figure  3.  Army  ACAT  IV  Management  Organizational  Structure 


by  individuals  within  the  MSCs  who  do  not  receive  a  formal  designation  as  Program  or 
Product  Manager.  For  programs  in  production,  these  ACAT  IV  managers  are  typically 
civilian  Weapons  System  Managers  at  the  MSC  headquarters.  For  those  programs  still 
in  development,  the  manager  is  typically  a  civilian  technical  expert  with  an  informal  title 
of  Project  Officer.  He  is  usually  located  at  a  Research,  Development,  and  Engineering 
Center  (RDEC)  that  is  subordinate  to  a  related  MSC. [Ref.  15] 

A  typical  example  of  an  ACAT  IV  program  would  be  the  Army’s  next  generation 
of  general  purpose  tentage,  the  Modular  General  Purpose  Tent  System.  This  effort  seeks 
to  apply  modem  materials  and  structures  to  replace  the  1950s  vintage  tentage  currently 
in  use  by  all  of  the  Services.  The  technological  risk  is  very  low,  total  development  costs 
will  be  less  than  $2  million,  and  the  requirement  is  not  threat  or  vulnerability  driven. 
This  logistics/quality-of-life  oriented  effort  certainly  does  not  warrant  the  oversight 
required  for  a  major  ACAT  I  or  II  weapon  system.  The  management  structure  supporting 
this  effort  includes  the  GS-12  civilian  Project  Officer  and  his  management  chain-of- 
command.  The  MDA  is  the  MSC,  Aviation  and  Troop  Command  (ATCOM),  Executive 
Director  for  Integrated  Materiel  Management.  [Ref.  16] 

C.  STREAMLINING  THE  PROCESS 

One  aspect  of  the  new  DoD  5000  series  that  did  not  exist  in  the  old  regulations 
is  that  the  Services  are  now  prohibited  from  regulatory  supplementation.  DoD  5000.2 
states: 

Unless  prescribed  by  statute  or  specifically  authorized  herein,  the 
policies  and  procedures  set  out  in  this  Instruction  shall  not  be 
supplemented  without  the  prior  approval  of  the  Under  Secretary  of 
Defense  for  Acquisition....  Implementation  directives,  instructions, 
regulations,  and  related  issuances  shall  be  kept  to  the  essential  minimum 
as  deemed  appropriate  by  the  DoD  Component  Acquisition  Executive. 

Copies  of  all  such  issuances  shall  be  provided  to  the  Director  of 
Acquisition  Policy  and  Program  Integration,  Office  of  the  Under  Secretary 
of  Defense  for  Acquisition  within  10  days  of  publication.  [Ref.  17] 
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This  effectively  ensures  that  the  Services  do  not  circumvent  the  goal  of  the  new 
regulation  by  instituting  Service-specific  implementation  policy.  Unfortunately,  it  also 
creates  an  inability  for  a  Service  to  implement  beneficial  across-the-board  policies 
without  OSD  approval.  Streamlining  the  procedures  and  processes  of  DoD  5000  for  all 
ACAT IV  programs,  for  instance.  DoD  5000  does,  however,  encourage  streamlining  or 
tailoring  of  the  process  on  a  project-by-project  basis.  DoD  5000.2  states: 

Tailoring  of  Acquisition  Procedures  and  Documentation.  The 
policies  and  procedures  described  in  this  Instruction  shall  apply  directly 
to  Acquisition  Category  I  programs  and  will  be  tailored  as  defined  in 
subsection  B.5.,  above,  for  acquisition  category  II,  III,  and  IV  programs 
subject  to  the  approval  of  the  milestone  decision  authority. [Ref.  18] 

Furthermore,  DoD  5000.1  requires  that  acquisition  strategies  and  program  plans 
be  streamlined  to  minimize  acquisition  time  and  life-cycle  costs,  consistent  with  the 
urgency  of  the  need  and  degree  of  technical  risk  involved.  Streamlining  becomes 
particularly  important  for  the  smaller,  ACAT  IV  programs  since  the  procedures  outlined 
in  DoD  5000  are  written  for  the  management  oversight  needs  of  the  largest  ACAT  I  and 
II  weapon  systems.  [Ref.  19] 

As  noted  in  the  above  excerpt  from  DoD  5000.2,  DoD  5000  gives  broad  power 
to  the  MDA  to  decide,  among  other  things,  whether  a  program  streamlining  plan  is 
acceptable.  This  power  was  most  recently  emphasized  by  the  Under  Secretary  of  Defense 
(Acquisition  and  Technology)  in  a  23  August  1994  memorandum  to  the  Service 
Secretaries  titled:  "Tailoring  Acquisition  Procedures  and  Documentation  for  Acquisition 
Category  II,  III  and  IV  Programs"  which  should  allow  for  consideration  of  more 
innovative  streamlining  methods  than  could  be  agreed  to  by  all  members  of  the 
acquisition  team  in  the  past.  The  memorandum  states: 

The  specific  form  or  number  of  program  documents  should  be  determined 
by  the  MDA.  As  long  as  tailoring  is  consistent  with  any  applicable 
statutory  requirements,  the  MDA  has  full  authority  to  reduce  or  eliminate 
any  procedures  or  documents  that  he  or  she  deems  unnecessary. [Ref.  20] 
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As  defined  in  the  Defense  Systems  Management  College’s  Risk  Management 
Concepts  and  Guidance  Manual,  streamlining  is  a  part  of  the  risk  management  decision 
making  process,  a  necessary  consideration  if  the  manager  is  to  optimize  his  project’s 
cost,  schedule  and  performance.  While  developing  a  program  plan  or  acquisition  strategy, 
as  well  as  during  the  life  of  a  program,  the  manager  must  consider  risk-related  trade-offs, 
some  of  which  involve  streamlining  considerations.  Risks  to  the  manager  are  all  rooted 
in  the  determination  to  deliver  a  specified  product  or  level  of  performance  at  a  specified 
time  for  a  specified  cost.  The  manager  may  fail  in  any  of  three  ways,  or  combination 
thereof.  The  product  may  not  be  up  to  the  performance  level  specified,  the  actual  costs 
may  be  too  high,  or  delivery  may  be  too  late.  According  to  the  Risk  Management 
Manual,  risk  can  be  categorized  into  five  facets  to  segment  and  manage  the  cost, 
schedule,  and  performance  issues  of  a  project.  These  risk  facets  are  technical 
performance,  supportability,  programmatic  (loss  of  funds,  availability  of  test  assets,  etc.), 
cost,  and  schedule.  If  streamlining  isn’t  considered,  projects  run  the  risk  of  being  too 
slow  to  fielding,  or  overly  costly  to  develop.  Streamlining  will  increase  risk,  but  the  goal 
of  any  effective  manager  is  to  balance  the  project’s  risk  with  the  cost,  schedule,  and 
performance  attributes  of  the  effort.  Thus,  streamlining  is  a  large  part  of  the  manager’s 
overall  risk  management  strategy.  [Ref.  21] 

Illustrative  examples  of  this  risk  management  process  as  it  relates  to  streamlining 
and  the  schedule,  performance,  and  cost  attributes  of  a  program  are:  (1)  Schedule:  An 
urgent  mission  need  for  a  capability,  either  because  of  an  increase  to  combat 
effectiveness  or  due  to  system  obsolescence,  can  lead  to  a  need  for  streamlining  of  the 
process  in  order  to  shorten  the  time  to  fielding  for  the  new  capability.  The  manager 
evaluates  this  requirement,  the  schedule  reduction  options  available,  and  the  risk 
associated  with  each;  (2)  Performance:  The  decision-maker  determines  that  a  program’s 
relatively  low  degree  of  technical  risk,  based  on  the  maturity  of  the  technology  being 
employed  and  the  complexity  of  the  system,  will  allow  for  skipping  of  milestone  events 
and  streamlining  of  the  program;  or  (3)  Cost:  The  decision  maker  considers  that 
streamlining  the  development  program  will  reduce  costs  in  both  development  and 
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operations  and  support  (where  the  capability  being  replaced  may  have  a  higher 
maintenance  cost  due  to  it  being  beyond  its  useful  life).  He  must  evaluate  cost  concerns 
with  an  acceptable  degree  of  risk. 

All  of  these  streamlining  considerations  are  based  on  trade-offs  of  costs  (e.g.,  lack 
of  availability  of  development  funds,  potential  O&S  cost  savings)  and  schedule  (e.g., 
urgent  need)  with  the  risk  of  technical  performance,  supportability,  or  programmatic 
failure  with  each  of  these  decisions.  To  properly  assess  these  considerations,  the  decision 
maker  requires  a  sound  risk  assessment  methodology  utilizing  data  concerning  the 
impacts  of  each  decision  option  and  the  key  factors  relating  to  each.  Key  factors  (utilized 
for  trade-offs  among  schedule,  cost,  and  risk)  would  be  attributes  such  as  technological 
maturity,  system  complexity,  urgency  of  need,  availability  of  funding  (both  for 
development  and  production),  and  availability  of  personnel  and  facilities  (for  oversight, 
testing,  etc.).  This  study  seeks  to  establish  this  assessment  data  and  key  factors  for  use 
by  AC  AT  IV  managers  in  conducting  risk  assessments  on  their  programs.  [Ref.  22] 

The  researcher’s  experience  and  information  from  the  MSC  POCs  identified  the 
streamlining  techniques  employed  for  ACAT  IV  projects  that  were  investigated  in  this 
research  effort.  Those  techniques  are  summarized  as  follows: 

1.  Combine  Milestones  0  and  I  (Skip  Concept  Exploration  (CE)  Phase) 

2.  Combine  Milestones  I  and  II  (Skip  Demonstration/Validation  (Dem/Val) 

Phase) 

3.  Combine  Milestones  I  and  III  (Skip  Dem/Val  and  Engineering  & 

Manufacturing  Development  (EMD)  Phases) 

4.  Combine  Dem/Val  Development  Test  (DT)  and  Operational  Test  (OT) 

5.  Conduct  concurrent  (overlapping)  Dem/Val  DT  and  OT 

6.  Combine  EMD  DT  and  Initial  Operational  Test  &  Evaluation  (IOT&E) 

7.  Conduct  concurrent  (overlapping)  EMD  DT  and  IOT&E 

8.  Conduct  Customer  Tests  in  lieu  of  a  formal  Development  Test  in  Dem/Val 

9.  Conduct  Customer  Tests  in  lieu  of  a  formal  Development  Test  in  EMD 

10.  Conduct  user  evaluation  in  lieu  of  formal  IOT&E  in  EMD  Phase 

1 1 .  Request  Milestone  Decision  Authority  delegate  Milestone  Decision  Review 

Chairmanship  to  RDEC  Management 

12.  Request  DA-Directed  Procurement  prior  to  Type  Classification-Standard 
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13.  Staff  Milestone  Decision  Package  prior  to  receipt  of  Independent 
Evaluation/ Assessment  Reports  (IER/IAR),  with  reports  expected  prior  to  the 
decision  meeting 

14.  Use  Interim  Assessment  or  Evaluation  Reports  to  support  a  Milestone 
Decision,  with  decision  contingent  upon  favorable  Final  Reports 

15.  Use  Independent  Evaluator  Briefing  in  lieu  of  written  reports  to  support  a 
Milestone  Decision,  with  written  report(s)  to  follow 

16.  Waive  requirement  for  a  Cost  and  Operational  Effectiveness  Analysis 
(COEA) 

17.  Waive  requirement  for  a  separate  System  Threat  Assessment  Report  (STAR) 

18.  Waive  requirement  for  a  separate  Logistics  Demonstration 

19.  Utilize  Test  Integration  Working  Group  (TIWG)  as  Electromagnetic 
Environmental  Effects  (E3)  Requirements  Board 

20.  Conduct  abbreviated  Logistics  Support  Analysis  and  Reporting  (LSA/LSAR) 

21.  Don’t  prepare  separate  Configuration  Management  Plan 

22.  Don’t  generate  Independent  Life  Cycle  Cost  Estimate 

23.  Obtain  Milestone  Decision  Authority  approval  to  solicit  a  contract  prior  to 
Milestone  Decision  Review 

24.  Utilize  a  single  contract  for  Dem/Val  and  EMD 

25.  Utilize  a  single  contract  for  EMD  and  First  Production 

26.  Conduct  production  First  Article  Tests  to  validate  development  deficiency 
corrections 

27.  Validate  technical  manuals/support  system  during  first  production  instead  of 
during  EMD 

Most  of  these  27  techniques  are  self-explanatory.  The  few  that  require  further 
explanation  are  described  as  follows: 

Techniques  8  and  9  -  Conduct  Customer  Tests  in  lieu  of  a  formal  DT  in 
Dem/Val  and  EMD.  Here,  Customer  Tests  are  defined  as  tests  conducted  by  the 
development  tester.  Test  and  Evaluation  Command  (TECOM),  that  are  based  solely  on 
the  direction  provided  by  the  materiel  developer.  For  these  techniques,  Customer  Tests 
do  not  refer  to  those  conducted  with  operational  testing  issues.  The  materiel  developer 
is  the  customer  of  TECOM  in  this  case.  This  differs  from  formal  DT  where  the  basis  for 
the  test  is  a  requirements  document.  Customer  tests  are  typically  used  in  early 
development  to  evaluate  concepts  and  they  are  usually  less  expensive  than  comparable 
DT  because  of  lower  instrumentation  and  oversight  requirements.  However,  the  results 
may  not  meet  the  needs  of  the  user. 
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Technique  10  -  Conduct  user  evaluation  in  lieu  of  formal  IOT&E  in  EMD  Phase. 
A  user  evaluation  is  defined  as  use  of  a  developmental  system  by  real  field  units  in  an 
actual  field  exercise,  with  non-intrusive  methods  of  data  collection  such  as  interviews 
following  the  exercise.  Formal  IOT&E,  on  the  other  hand,  is  defined  as  use  of  a 
developmental  system  by  representative  soldiers  in  a  field  exercise  created  specifically 
to  evaluate  the  performance  of  the  new  system.  IOT&E  is  substantially  more  costly  than 
a  user  evaluation,  but  it  can  also  result  in  substantially  more  useful  data  on  the 
performance  of  a  developmental  system. 

Technique  12  -  Request  DA-Directed  Procurement  prior  to  Type  Classification- 
Standard.  In  cases  where  fielding  of  a  developmental  system  is  urgently  required  and  the 
development  process  has  not  been  completed,  DA  has  the  option  to  direct  the  immediate 
procurement  of  the  item.  This  would  then  be  followed  by  completion  of  formal 
development  and  the  retrofitting  of  any  required  changes  into  the  previously  fielded 
systems. 

Technique  19  -  Utilize  Test  Integration  Working  Group  (TIWG)  as 
Electromagnetic  Environmental  Effects  (E3)  Requirements  Board.  The  requirement  for 
an  E3  Requirements  Board  for  all  Army  development  programs  was  recently  directed  by 
AMC.  Since  the  agencies  and  individuals  necessary  for  both  the  E3  Board  and  the 
already  existing  TIWG  are  almost  identical,  combining  of  the  two  would  save  on  the 
number  of  separate  committees  involved  in  oversight  of  a  program. 

Technique  26  -  Conduct  production  First  Article  Tests  to  validate  development 
deficiency  corrections.  For  most  development  programs,  any  serious  system  deficiencies 
uncovered  during  EMD  must  be  corrected,  and  the  corrections  validated  before  the  Army 
is  willing  to  enter  production.  The  validation  could  consist  of  an  extra  evaluation/test 
prior  to  conducting  the  Milestone  III  Decision  Review.  First  Article  Testing  validates  the 
adequacy  of  production  units  before  allowing  full  rate  production  to  begin,  using  the  first 
units  produced  by  the  production  contractor.  It  usually  retests  many  of  the  same  functions 
as  would  be  required  to  validate  development  deficiency  corrections.  Therefore,  if  the 
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MDA  is  willing  to  accept  the  Milestone  III  Decision  prior  to  the  validation  of  the 
deficiency  corrections,  they  could  be  validated  during  FAT. 

A  hypothetical  example  of  streamlining  of  an  ACAT  IV  program  would  be  as 
follows:  The  manager  may  consider  the  technical  risk  of  the  system  that  is  to  be 
developed  to  be  low  enough  to  allow  for  a  combined  Milestone  I  and  II,  moving  the 
project  directly  from  Concept  Exploration  (CE)  into  the  Engineering  and  Manufacturing 
Development  (EMD)  phase.  The  risk  of  this  approach  is  that  the  system  will  not  be 
sufficiently  mature  for  EMD,  resulting  in  rework  and  delays  and/or  less  than  desired 
performance.  But  the  manager  must  balance  this  risk  with  the  need  to  field  the  new 
system  by  a  prescribed  date  (for  instance,  driven  by  the  end  of  the  useful  life  of  the 
existing  system).  Streamlining  provides  the  best  management  plan  given  the  urgency  of 
the  schedule  and  the  relatively  low  risk. 

D.  CHAPTER  SUMMARY 

The  new  DoD  5000  series  of  acquisition  management  regulations  concentrates  on 
the  management  oversight  required  for  large  ACAT  I  programs.  The  regulations  outline 
the  processes  required  of  ACAT  I  managers,  and  encourages  tailoring,  or  streamlining 
of  the  process  by  ACAT  II,  III,  and  IV  managers,  while  not  allowing  for  across-the- 
board  Service-level  policies.  Individual  managers  need  to  streamline  to  effectively 
manage  their  program’s  risk,  but  they  lack  key  data  necessary  for  conducting  proper  risk 
assessments.  This  study  seeks  to  provide  this  missing  risk  assessment  data. 
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ffl.  METHODOLOGY 


The  methodology  utilized  for  this  effort  was  a  management  problem  analysis  using 
both  quantitative  and  qualitative  techniques.  The  qualitative  techniques  involved  the  use 
of  personal  and  telephonic  interviews  to  determine  the  opinions  of  the  acquisition  policy 
Subject  Matter  Experts  from  each  of  the  AMC  Commands  concerning  the  validity  and 
breadth  of  the  techniques  to  be  investigated.  The  information  from  this  process  was 
assimilated  into  a  broad-based  questionnaire  that  was  distributed  Army- wide  to  all  AC  AT 
IV  development  managers,  with  questions  that  required  numeric  and  essay  responses. 
The  numeric  responses  were  quantitatively  analyzed,  and  the  essay  responses  were  used 
to  find  elaboration/causal  relationships  for  specific  responses  and/or  trends  observed  in 
the  data. 

A.  RESEARCH  PROCESS  STEPS 

1.  Telephonic  interviews  were  conducted  with  individuals  within  AMC,  the 
Department  of  Army  Headquarters,  and  the  AMC  MSCs  to  develop  a  list  of 
MSC/RDEC  acquisition  policy  points  of  contact  (POCs)  for  ACAT IV  programs. 
The  individuals  identified  were  either  the  Command’s  overall  acquisition  policy 
POC,  or  a  senior  manager  charged  with  management  of  all  ACAT  IV  programs 
for  that  Command.  The  individuals  are  listed  in  Appendix  A. 

2.  Telephonic  interviews  were  conducted  to: 

a.  Develop  a  valid  list  of  streamlining  techniques  used  by  ACAT  IV  managers, 
for  use  as  a  basis  for  questionnaires  to  ACAT  IV  managers. 

b.  Identify  ACAT  IV  project  managers’  names  and  addresses  to  survey. 
Development  of  this  list  was  necessary  because  ACAT  IV  programs  are  not  all 
tracked  by  AMC.  Therefore,  the  only  way  to  obtain  a  comprehensive  list  was  to 
call  into  each  Command  and  search  out  an  individual  or  individuals 
knowledgeable  on  the  subject  who  had  access  to  the  names  and  addresses  of  the 
ACAT  IV  managers.  A  total  of  seventy-seven  individuals  were  identified  as 
ACAT  IV  managers.  For  all  Commands  except  the  Missile  Command  (MICOM) 
and  the  Aviation  Applied  Technology  Directorate  (AATD)  of  the  Aviation  and 
Troop  Command,  specific  names  and  addresses  were  identified.  For  MICOM  and 
AATD,  since  only  one  senior  manager  within  each  Command  was  in  charge  of 
all  ACAT  IV  programs,  the  POCs  noted  above  agreed  to  distribute  the 
questionnaires. 

3.  Using  the  streamlining  techniques  previously  listed,  and  the  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  Questionnaire 
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Construction  Manual,  an  appropriate  survey  questionnaire  for  the  study  was 
constructed.  See  Section  B  for  the  specifics  of  the  questionnaire  development 
process  and  Appendix  B  for  a  copy  of  the  questionnaire.  [Ref.  23] 

4.  Draft  questionnaires  were  provided  to  the  MSC/RDEC  acquisition  policy  POCs 
for  their  review  and  comment.  Since  these  individuals  are  their  Command’s 
acquisition  policy  "experts",  their  responses  helped  refine  the  document  and 
served  as  a  check  that  the  questionnaire  was  complete  and  understandable  before 
surveying  their  AC  AT  IV  managers.  The  POCs  were  not  surveyed  further. 

5.  The  questionnaires  were  sequentially  numbered  for  control  purposes  and 
mailed  to  each  of  the  AC  AT  IV  managers  that  were  previously  identified.  The 
mailing  included  a  personally  addressed  and  individually  signed  cover  letter 
requesting  participation,  stating  the  purpose  of  the  study,  and  articulating  the 
voluntary,  confidential  and  not-for-attribution  nature  of  the  study.  Additionally, 
each  package  contained  a  return  addressed,  postage  paid,  Government  reply 
envelope.  Four  weeks  were  allowed  for  responses. 

6.  Finally,  the  data  was  compiled  and  analyzed  as  discussed  in  Chapter  V. 


B.  QUESTIONNAIRE  DEVELOPMENT 

A  quantitative  questionnaire  with  provisions  for  subjective  comments,  reproduced 
in  Appendix  B,  was  prepared  as  the  survey  instrument.  It  is  structured  in  two  parts.  The 
first  part  focuses  on  the  size  of  the  projects  managed  by  the  respondent,  to  allow  for 
categorization  of  responses  by  size.  The  second  part  addresses  the  respondents  opinions 
regarding  DoD  5000  streamlining  techniques.  The  questionnaire  structure  chosen  was  an 
ordinal,  Likert  scale,  with  compound  questions:  a  binary  set  and  two  five  point  adjectival 
questions.  The  following  scale  was  employed  for  the  adjectival  questions  regarding  cost 
savings  and  schedule  reduction  potential.  This  non-linear  scale  was  chosen  to  be  able  to 
differentiate  among  the  levels  of  savings  for  the  lowest,  as  well  as  the  highest,  saving 
techniques: 


Rating  1 

Cost  Savings  $0-$100 

Schedule  Reduction  0-7  days 


2  3 

$100-$1000  $1K-$10K 

1-4  wks  1-6  mos 


4  5 

$10K-$100K  >$10CK 
6-18  mos  >18  ms 
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From  the  referenced  questionnaire  manual,  these  selections  best  fit  the  type  of 
questions  and  data  being  addressed  in  this  research  effort.  For  example,  the  reference 
states  that  ordinal  Likert  scales  are  the  preferred  and  most  widely  used  type  when 
multiple  choice  responses  are  required  and  the  distance  between  each  scale  point  isn’t 
assumed  to  be  equal.  In  addition,  it  is  the  preferred  scale  when  the  categories  are 
mutually  exclusive  and  exhaustive,  as  the  list  of  streamlining  initiatives  should  be.  Five 
point  scales  are  the  most  common  and  within  the  range  of  breakouts  that  yields  roughly 
equal  responses.  The  scale  will  be  fully  labeled,  as  this  is  noted  in  the  reference  to 
provide  less  skewed  results.  This  should  provide  an  understandable  and  statistically  useful 
range  of  options.  [Ref.  24] 

The  questionnaire  addressed  the  research  questions  as  outlined  in  the  following 
dendritic  process: 

Subsidiary  Research  Question  1:  How  do  Army  Commands  streamline  ACAT  IV 
programs  differently? 

Criteria:  For  each  Command,  what  are  the  most  frequently  used  streamlining 
techniques? 

Measure  of  Effectiveness:  For  each  Command,  what  techniques  are  used? 

Data  Requirement:  Techniques  used  by  each  Command. 

Data  Source:  Survey  Question  a. 

Measure  of  Effectiveness:  For  each  Command,  what  is  the  frequency  of  use  of 

each  technique? 

Data  Requirement:  Number  of  times  each  technique  is  used  by  each 

Command. 

Data  Source:  Survey  Question  a. 
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Criteria:  What  factors  cause  each  Command  to  use  certain  techniques  (as  compared  to 
others)? 

Measure  of  Effectiveness:  For  the  AC  AT  IV  managers  in  each  Command,  why 
was  a  technique  chosen? 

Data  Requirement:  Manager  comments  on  factors  leading  to  a  decision. 
Data  Source:  Survey  Question  a.  comments. 

Subsidiary  Research  Question  2:  How  do  the  various  streamlining  techniques  compare 
with  respect  to  program  cost  savings? 

Criteria:  For  each  technique,  what  is  the  program  cost  savings? 

Measure  of  Effectiveness:  For  each  size  program  what  is  the  distribution  of  cost 
savings  ratings  among  the  streamlining  techniques? 

Data  Requirement:  Ratings  of  cost  savings  associated  with  each 
technique. 

Data  Source:  Survey  Question  b. 

Criteria:  What  factors  cause  each  manager  to  utilize  certain  techniques  (as  compared  to 
others)? 

Measure  of  Effectiveness:  For  each  manager,  why  was  a  rating  chosen? 

Data  Requirement:  Manager  comments  on  factors  leading  to  a  decision. 
Data  Source:  Survey  Question  b.  comments. 

Subsidiary  Research  Question  3:  How  do  the  various  streamlining  techniques  compare 
with  respect  to  program  schedule  reduction? 

Criteria:  For  each  technique,  what  is  the  program  schedule  reduction? 

Measure  of  Effectiveness:  For  each  size  program  what  is  the  distribution  of 
schedule  reductions  ratings  among  the  streamlining  techniques? 

Data  Requirement:  Ratings  of  schedule  reductions  associated  with  each 
technique. 

Data  Source:  Survey  Question  c. 
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Criteria:  What  factors  cause  each  manager  to  utilize  certain  techniques  (as  compared  to 
others)? 

Measure  of  Effectiveness:  For  each  manager,  why  was  a  rating  chosen? 

Data  Requirement:  Manager  comments  on  factors  leading  to  a  decision. 

Data  Source:  Survey  Question  c.  comments. 

The  survey  conducted  utilizing  the  questionnaire,  as  part  of  this  study’s 
methodology,  resulted  in  quantitative  and  narrative  data  concerning  the  streamlining 
techniques  being  investigated.  In  the  next  chapter  the  data  will  be  reviewed,  tabulated  and 
presented  for  analysis. 
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IV.  DATA  PRESENTATION 


In  support  of  this  thesis,  74  questionnaires  were  distributed  to  the  ACAT  IV 
development  managers  at  various  U.S.  Army  Commands  to  inquire  about  their  use  of 
program  streamlining  techniques  for  ACAT  IV  development  programs.  Initially  there 
were  16  responses.  After  follow-up  inquiries,  it  was  discovered  that  35  individuals  were 
inappropriate  survey  subjects,  so  the  number  of  potential  respondents  was  reduced  to  39 
and  a  final  total  of  19  responses  were  obtained.  Some  Commands  provided  limited  or  no 
responses. 

The  two  Commands  that  did  not  respond  were  CECOM  and  MICOM.  Since  these 
two  cases  of  nonresponsiveness  appeared  to  represent  systemic  problems,  the  Command 
ACAT  IV  acquisition  policy  POCs  (Appendix  A)  were  queried  about  the  lack  of 
responses,  instead  of  directly  contacting  individual  ACAT  IV  managers.  The  discussions 
with  these  POCs  revealed  a  difference  with  the  way  smaller-sized  development  programs 
are  managed  for  the  commodities  associated  with  CECOM  and  MICOM.  This  difference 
was  not  readily  apparent  during  earlier  discussions.  Specifically,  both  Commands  use 
managers  who  are  not  certified  Program  Managers  to  conduct  the  early  development 
phase  of  programs,  which,  as  pointed  out  earlier,  automatically  categorizes  them  as 
ACAT  IV.  What  is  unique  to  these  two  Commands  is  that  all  of  their  programs  transition 
to  ACAT  III,  or  higher  status  early  in  development  by  having  management  responsibility 
transferred  to  a  certified  Program  Manager  (PM)  within  the  Program  Executive  Officer 
(PEO)  chain-of-command.  This  also  happens  for  some  but  not  all  programs  at  the  other 
AMC  Commands.  Therefore,  the  ACAT  IV  managers  within  CECOM  and  MICOM  do 
not  get  very  involved  in  the  DoD  5000  policies,  most  of  which  are  not  applicable  early 
in  the  development  process,  and  the  managers  had  no  experience  base  with  which  to 
respond  to  the  survey.  The  CECOM  POC  stated: 

The  survey  you  supplied  seems  to  be  directed  more  to  the  type  of 
acquisition  done  at  Natick.  Here  at  CECOM,  a  program  goes  through  the 
early  stages  of  development,  funded  by  6.2,  6.3  R&D  funds.  When  a 
program  is  ready  for  6.4  funding  (the  EMD  phase),  it  transitions  to  a  PEO 
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portfolio....  ACAT  IV,  by  definition,  means  the  system  is  managed  by 
other  than  a  board  selected  Program  Manager.  Therefore,  a  system  that 
starts  as  an  ACAT  IV  can  then  become  an  ACAT  III  when  it  transfers  to 
a  PM  under  a  PEO. 

Given  this  difference,  these  two  Commands  fall  outside  of  the  scope  of  the  survey 
since  it  keyed  on  ACAT  IV  development  managers  and  the  policies  of  DoD  5000. 
Thereby,  the  number  of  potential  survey  respondents  is  appropriately  reduced  to  39. 
However,  the  difference  in  management  philosophies  between  these  two  Commands  and 
the  other  four  is  of  interest,  and  the  Analysis  Chapter  includes  a  comparison  based  on 
the  information  revealed  in  the  study  and  the  researcher’s  experience. 

Follow-up  discussions  with  non-responsive  managers  from  the  other  Commands 
revealed  a  number  of  points  regarding  the  questionnaire  and  the  researcher’s 
assumptions.  Assumptions  concerning  the  managers’  willingness  to  participate  and  their 
knowledge  of  the  DoD  5000  development  process  were  not  valid  for  all  ACAT  IV 
managers.  Also,  a  number  of  managers  did  not  consider  themselves  qualified  to  respond 
due  to  their  lack  of  experience  with  the  entire  process  (which  can  take  long  enough  that 
many  managers  have  not  experienced  all  or  much  of  it  in  their  careers),  and/or  lack  of 
knowledge  of  the  processes  being  proposed  for  streamlining.  Some  of  the  individuals 
contacted  were  not  familiar  with  the  DoD  5000  terminology,  and  this  created  a  barrier 
to  their  understanding  of  the  proposed  streamlining  techniques. 

Given  the  above,  the  data  that  was  received  is  deemed  to  be  valid.  The 
respondents  are  still  assumed  to  be  knowledgeable  and  any  conclusions  drawn  valid.  A 
limitation  is  created  as  it  will  be  difficult  to  establish  statistical  significance  for  the 
results,  since  the  quantity  of  observations  is  lower  than  expected.  Therefore,  the  analysis 
will  be  conducted  on  a  qualitative,  rather  than  predominately  quantitative,  basis. 

A.  DATA  TABULATION 

As  the  questionnaire  were  received  from  the  respondents,  the  data  was  loaded  in 
two  databases.  The  quantitative  responses  to  the  survey  were  tabulated  in  a  Minitab 
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database  and  the  narrative  responses  were  tabulated  by  question  number  in  a  WordPerfect 
text  database.  The  raw  data  is  available  in  Appendices  C  and  D  for  the  quantitative  and 
narrative  data  respectively. 

B.  DATA  PRESENTATION 

For  the  first  research  sub-question  (How  do  Army  Commands  streamline  ACAT 
IV  programs  differently?),  the  Command-by-Command  comparison,  the  quantitative 
responses  are  summarized  in  Table  1 .  For  each  Command  that  was  queried  in  the  survey, 
the  table  depicts  the  total  number  of  respondents,  as  well  as  the  number  and  proportion 
of  the  respondents  who  responded  positively  regarding  their  awareness  of  each  of  the 
streamlining  techniques.  Note  the  low  number  (two)  of  respondents  from  both  AATD  and 
ERDEC,  as  well  as  the  lack  of  responses  from  AATD,  ERDEC,  and  TARDEC  (zeros 
in  the  ”#  Yes"  and  "  %  Yes"  columns)  for  a  number  of  techniques.  One  aspect  of  the 
Analysis  Chapter  will  be  investigation  of  each  of  these  non-response  areas,  to  ascertain 
whether  the  Commands  were  aware  of  the  technique  or  not. 

For  the  second  and  third  research  sub-questions  (How  do  the  various  streamlining 
techniques  compare  with  respect  to  program  cost  savings?)  and  (How  do  the  various 
streamlining  techniques  compare  with  respect  to  program  schedule  reduction?),  the 
Technique-by-Technique  comparisons,  the  quantitative  responses  are  summarized  in 
Table  2.  The  five  categories  for  total  program  RDTE  cost,  and  the  seven  categories  for 
development  durations,  established  in  the  survey  were  consolidated  in  the  table  into  two 
categories  (small  and  large  programs),  to  provide  clarity  for  evaluation.  This  was 
necessary  because  the  number  of  categories  established  during  questionnaire  development 
became  inappropriate,  due  to  the  small  number  of  survey  responses  in  each  category. 
Large  programs  were  defined  for  purposes  of  the  analysis  as  those  requiring  $7  million 
or  more  of  RDTE  funding  (for  the  cost  comparisons)  and  3  years  or  more  to  complete 
development  (for  the  schedule  comparisons).  For  each  size  program,  the  table  depicts  the 
total  number  of  respondents  for  both  cost  savings  and  schedule  reduction,  as  well  as  the 
number  of  respondents  who  replied  regarding  each  technique  and  an  average  of  the 
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ACAT  IV  Streamlining  Techniques 
Frequency  of  Use  by  Command 

Command 

Natick 

AATD 

ERDEC 

RSI 

No.  of 

Respondents 

9 

2 

2 

6 

Streamlining 

Technique 

#  Yes 

%  Yes 

#  Yes 

%  Yes 

#  Yes 

%  Yes 

1 

7 

0.8 

2 

1.0 

1 

0.5 

4 

0.7 

2 

9 

1.0 

2 

1.0 

2 

1.0 

3 

0.5 

3 

6 

0.7 

0 

0.0 

2 

1.0 

2 

0.3 

4 

9 

1.0 

0 

0.0 

2 

1.0 

4 

0.7 

5 

5 

0.6 

2 

1.0 

2 

1.0 

3 

0.5 

6 

5 

0.6 

1 

0.5 

2 

1.0 

1 

0.2 

7 

3 

0.3 

2 

1.0 

2 

1.0 

2 

0.3 

8 

5 

0.6 

1 

0.5 

2 

1.0 

0 

0.0 

9 

3 

0.3 

0 

0.0 

2 

1.0 

0 

0.0 

10 

6 

0.7 

2 

1.0 

2 

1.0 

i 

0.2 

11 

7 

0.8 

1 

0.5 

0 

0.0 

3 

0.5 

12 

4 

0.4 

1 

0.5 

0 

0.0 

0 

0.0 

13 

8 

0.9 

2 

1.0 

2 

1.0 

4 

0.7 

14 

7 

0.8 

2 

1.0 

2 

1.0 

4 

0.7 

15 

5 

0.6 

1 

0.5 

1 

0.5 

3 

0.5 

16 

7 

0.8 

2 

1.0 

1 

0.5 

2 

0.3 

17 

6 

0.7 

1 

0.5 

2 

1.0 

3 

0.5 

18 

8 

0.9 

0 

0.0 

2 

1.0 

2 

0.3 

19 

4 

0.4 

2 

1.0 

1 

0.5 

2 

0.3 

20 

3 

0.3 

1 

0.5 

2 

1.0 

2 

0.3 

21 

3 

0.3 

1 

0.5 

2 

1.0 

0 

0.0 

22 

2 

0.2 

0 

0.0 

1 

0.5 

1 

0.2 

23 

6 

0.7 

1 

0.5 

1 

0.5 

1 

0.2 

24 

8 

0.9 

1 

0.5 

2 

1.0 

3 

0.5 

25 

7 

0.8 

1 

0.5 

2 

1.0 

1 

0.2 

26 

6 

0.7 

2 

1.0 

2 

1.0 

2 

0.3 

27 

4 

0.4 

2 

1.0 

2 

1.0 

3 

0.5 

Table  1.  Frequency  of  Use  by  Command 
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Projected  Savings  by  Program  Size 
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Table  2.  Projected  Savings  by  Program  Size 


savings  ratings  provided  by  these  respondents.  In  order  to  qualitatively  evaluate  the 
results  of  the  survey,  the  ratings  of  the  individual  respondents  were  averaged  to:  (1) 
obtain  small  to  large  program  trends  in  the  savings  associated  with  the  various 
techniques,  and  (2)  establish  which  techniques  have  the  very  highest  and  lowest  savings 
(in  terms  of  cost  and  schedule)  for  small  and  large  programs.  The  average  ratings  do  not 
represent  expected  savings.  The  scale  is  not  linear  and  the  response  rates  are  too  low  to 
provide  confidence  in  the  point  estimates.  The  rating  scale  used  in  the  survey  is  provided 
at  the  bottom  of  the  table  for  reference. 

The  following  Analysis  Chapter  utilizes  the  data  in  Tables  1  and  2  to  investigate 
differences  between  the  usage  levels  of  Natick  and  a  combination  of  the  other 
Commands.  In  addition,  differences  between  the  cost  savings  and  schedule  reduction 
potential  of  small  and  large  programs  are  analyzed. 
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V.  ANALYSIS 


This  analysis  is  based  on  comparisons  of  the  most  prevalent  streamlining 
techniques  and  those  that  had  the  highest  and  lowest  savings,  with  respect  to  cost  and 
schedule.  The  lowest  saving  techniques  are  being  evaluated  because  a  number  of  them 
were  both  low  in  savings  and  high  in  use.  This  indicates  that  they  may  be  valuable  for 
other  reasons,  or  their  implementation  is  so  easy  as  to  be  routine.  The  narrative 
responses  are  used  to  provide  "cause-and-effect"  reasons  for  similarities  and  differences 
among  Commands  and  among  techniques  (in  terms  of  their  frequency  of  use  and  the 
expected  level  of  cost  savings  and  schedule  reduction). 

A  review  of  the  narrative  comments  regarding  three  of  the  techniques  led  to  their 
subsequent  elimination  for  further  analysis.  The  three  techniques  are 

Technique  8.  Conduct  Customer  Tests  in  lieu  of  a  formal  Development 

Test  in  Dem/Val 

Technique  9.  Conduct  Customer  Tests  in  lieu  of  a  formal  Development 

Test  in  EMD 

Technique  23.  Obtain  Milestone  Decision  Authority  approval  to  solicit  a 

contract  prior  to  Milestone  Decision  Review 

The  first  problem,  regarding  Techniques  8  and  9  (Conducting  Customer  Tests  in 
lieu  of  formal  DT)  concerned  varying  definitions  for  the  term  "customer  test".  The 
intended  definition  was  a  TECOM  test  conducted  without  formal  documentation  and 
planning,  where  the  materiel  developer  is  the  customer.  Many  respondents  from 
Commands  other  than  Natick  responded  as  if  "customer  testing"  meant  "user  testing", 
which  was  addressed  in  Technique  10  (Conduct  user  evaluation  in  lieu  of  formal  IOT&E 
in  EMD  phase).  The  varying  interpretations  of  the  definition  of  customer  testing  makes 
any  comparisons  based  on  these  techniques  invalid.  For  Technique  23  (Obtain  MDA 
approval  to  solicit  a  contract  prior  to  Milestone  Decision  Review)  the  narrative  comments 
from  the  survey  participants  indicated  that  some  respondents  misread  the  technique  and 
they  addressed  it  as  if  it  proposed  awarding  the  contract,  vice  soliciting  it.  The  question 
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did  not  intend  to  address  the  award  of  a  contract  prior  to  the  Review.  Given  this,  the 
responses  are  not  clearly  differentiated  and  the  information  is  not  clear.  Therefore,  the 
analysis  ignores  Techniques  8,  9,  and  23. 

A.  COMMAND-BY-COMMAND  COMPARISON 

1.  Command  Awareness  of  Techniques 

The  first  Measure  of  Effectiveness  addressed  the  question  of  whether  all  the 
Commands  were  aware  of  all  of  the  techniques.  From  the  data  in  Table  1,  there  are 
seven  techniques  that  were  not  considered  by  any  of  the  respondents  for  at  least  one 
Command. 


Technique  3. 

Technique  4. 

Technique  11. 

Technique  12. 

Technique  18. 
Technique  21. 
Technique  22. 


Combine  Milestones  I  and  III  (Skip  Dem/Val  and 
Engineering  &  Manufacturing  Development  (EMD)  Phases) 
Combine  Dem/Val  Development  Test  (DT)  and  Operational 
Test  (OT) 

Request  Milestone  Decision  Authority  delegate  Milestone 
Decision  Review  Chairmanship  to  RDEC  Management 
Request  DA-Directed  Procurement  prior  to  Type 
Classification-Standard 

Waive  requirement  for  a  separate  Logistics  Demonstration 
Don’t  prepare  separate  Configuration  Management  Plan 
Don’t  generate  Independent  Life  Cycle  Cost  Estimate 


Because  the  questionnaires  were  targeted  to  specific  individuals,  the  non-responses 
concerning  these  techniques  may  not  indicate  that  the  Command  was  not  aware  of  the 
technique,  but  only  that  the  individuals  responding  were  not.  In  addition,  some 
individuals  who  did  not  check  the  survey  block  to  indicate  that  they  were  aware  of  a 
technique  did,  however,  provide  narrative  comments  regarding  the  same  technique.  Given 
this,  a  review  of  the  survey  narrative  comments  for  these  seven  techniques  revealed  the 
following  specific  information: 

For  Techniques  3  (Combine  Milestones  I  and  III)  and  4  (Combine  Dem/Val  DT 
and  OT),  AATD  did  not  have  any  positive  responses.  However,  evaluation  of  their 
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narrative  responses  revealed  that  they  were  aware  of  the  techniques  and  that  they  have 
sometimes  used  them.  Specific  comments  were: 

Technique  3  -  Off-the-shelf  or  low  risk  technology  are  ideally  suited  for 
combining  I  and  III  for  an  AC  AT  IV  program. 

Technique  4  -  Off-the-shelf  items  should  not  need  very  much  DT/OT 
testing  because  of  the  low  technology  risks. 

For  Techniques  11  (Request  MDA  delegate  Milestone  Decision  Review 
Chairmanship  to  RDEC  management),  12  (Request  DA-Directed  Procurement  prior  to 
Type  Classification-Standard),  21  (Don’t  prepare  separate  Configuration  Management 
Plan),  and  22  (Don’t  generate  Independent  Life  Cycle  Cost  Estimate)  various  Commands 
did  not  have  any  positive  responses,  but  their  narrative  comments  revealed  an  awareness 
of  the  technique.  In  addition,  their  comments  expressed  disagreement  with  use  of  the 
technique  as  the  reason  they  did  not  respond  positively  to  the  survey  question.  Comments 
included: 

Technique  11.-  Like  having  the  General  Officer  sign  it. 

Technique  21.  -  Configuration  Management  Plan  must  be  specific  to 
parties  involved. 

For  Technique  18  (Waive  requirement  for  a  separate  Logistics  Demonstration) 
AATD  did  not  have  any  positive  responses.  A  review  of  their  narrative  responses 
indicated  agreement  with  the  intent  of  the  technique,  but  there  was  no  indication  of 
whether  or  not  they  were  aware  of  the  idea  prior  to  the  survey. 

In  general,  for  the  seven  techniques  that  were  not  considered  by  any  of  the 
respondents  from  a  particular  Command,  all  of  the  individuals  who  did  not  respond 
provided  narrative  comments  and  in  all  but  one  case,  the  Command  was  aware  of  the 
technique.  Negative  responses  were  due  to  disagreement  with  the  use  the  technique,  not 
lack  of  awareness  of  the  technique.  The  one  technique  where  the  narrative  responses  left 
it  unclear  as  to  whether  AATD  was  aware  of  the  technique  (Waiving  the  Logistics 
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Demonstration),  remains  unknown.  However,  out  of  five  Commands  and  25  techniques 
(125  categories)  only  one  category  remained  unresolved  as  to  the  Command’s  awareness 
of  a  technique. 

All  of  the  Commands  are  aware  of  all  of  the  identified,  available  techniques.  Any 
methods  of  cross-fertilization  transmission  are  unknown.  They  may  be  independently 
concluding  the  same  things,  or  they  could  be  receiving  suggestions  from  acquisition  team 
members  who  are  members  of  teams  from  different  Commands  (such  as  TRADOC 
schools,  TECOM,  AMC  HQ,  etc.).  Regardless  of  the  method  being  employed,  the 
researcher’s  concern  that  cross-fertilization  was  not  taking  place  and  certain  Commands 
were  unaware  of  specific  streamlining  techniques  has  proven  to  be  invalid. 

2.  Highest  Usage  Techniques 

To  evaluate  the  second  Measure  of  Effectiveness,  comparing  usage  levels  across 
techniques  and  Commands,  the  Natick  responses  were  compared  to  those  of  all  other 
Commands  combined  (labeled  "non-Natick”  throughout  this  document),  see  Table  3.  As 
a  result  of  combining  non-Natick  responses,  the  analysis  of  Natick  operations  is  more 
thorough  than  any  other  Command.  This  was  felt  to  be  the  most  useful  approach  to  data 
analysis,  due  to  the  limited  responses  by  the  non-Natick  Commands.  First,  an  overall 
comparison  of  Natick  to  non-Natick  responses  will  be  conducted,  followed  by  a  more  in- 
depth  evaluation  of  the  techniques  with  the  highest  response  frequencies  for  Natick  and 
non-Natick.  As  a  result,  a  number  of  interesting  correlations  are  revealed. 

Overall  Comparison  -  Figure  4  compares  the  frequency  of  use  established  for 
each  technique  for  Natick  and  non-Natick  respondents.  The  chart  reveals  general 
agreement  between  the  two  groups,  with  a  consistently  slightly  higher  usage  for  the 
Natick  group.  One  possible  reason  for  the  consistent  difference  may  be  a  bias  created 
during  the  establishment  of  the  streamlining  techniques  for  the  study.  The  researcher  is 
from  Natick  and  a  number  of  the  techniques  listed  in  the  survey  were  posed  as  a  direct 
result  of  acquisition  policy  issues  recently  discussed  at  Natick,  and,  therefore,  the  Natick 
respondents  may  have  been  more  familiar  with  the  wording  in  the  survey. 
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ACAT IV  Streamlining  Techniques 

Frequency  of  Use 

Natick 

Other  Commands' 
Total 

No.  of 

Respondents 

9 

10 

Streamlining 

Technique 

#  Yes 

%  Yes 

#  Yes 

%  Yes 

1 

7 

0.8 

7 

0.7 

2 

9 

1.0 

7 

0.7 

3 

6 

0.7 

4 

0.4 

4 

9 

1.0 

6 

0.6 

5 

5 

0.6 

7 

0.7 

6 

5 

0.6 

4 

0.4 

7 

3 

0.3 

6 

0.6 

8 

5 

0.6 

3 

0.3 

9 

3 

0.3 

2 

0.2 

10 

6 

0.7 

5 

0.5 

11 

7 

0.8 

4 

0.4 

12 

4 

0.4 

1 

0.1 

13 

8 

0.9 

8 

0.8 

14 

7 

0.8 

8 

0.8 

15 

5 

0.6 

5 

0.5 

16 

7 

0.8 

5 

0.5 

17 

6 

0.7 

6 

0.6 

18 

8 

0.9 

4 

0.4 

19 

4 

0.4 

5 

0.5 

20 

3 

0.3 

5 

0.5 

21 

3 

0.3 

3 

0.3 

22 

2 

0.2 

2 

0.2 

23 

6 

0.7 

3 

0.3 

24 

8 

0.9 

6 

0.6 

25 

7 

0.8 

4 

0.4 

26 

6 

0.7 

6 

0.6 

27 

4 

0.4 

7 

0.7 

Table  3.  Frequency  of  Use,  Natick  vs.  Other  Commands  Total 
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Figure  4.  Frequency  of  Use  by  Command 


The  following  is  a  review  of  each  of  the  techniques  with  the  highest  differences, 
analyzing  the  differences  between  the  two  groups: 

For  Technique  4  (Combine  Dem/Val  DT  and  OT)  the  non-Natick  respondents 
expressed  lower  usage  for  three  reasons:  (1)  they  disagreed  with  the  use  of  the  technique 
because  of  its  high  risk  and  they  felt  that  independent  operational  tests  were  necessary 
to  remove  developer  biases,  (2)  they  considered  it  too  difficult  to  accomplish,  citing 
problems  with  obtaining  agreement  from  testers  and  evaluators,  and  (3)  commodity 
specific  concerns  (such  as  the  artillery  ammunition  not  typically  requiring  OT  and  the 
man-firing  of  munitions  prior  to  accomplishing  significant  DT).  Note  that  non-Natick 
usage  was  still  fairly  high  ( it  was  a  high  usage  technique  for  both  groups). 

Technique  1 1  (Request  MDA  delegate  Milestone  Decision  Review  Chairmanship 
to  RDEC  management)  is  broadly  applied  at  Natick.  The  non-Natick  respondents 
expressed  lower  usage  for  two  reasons:  (1)  they  disagreed  with  the  use  of  the  technique 
because  they  preferred  that  a  General  Officer  sign  the  minutes/decision,  or  because  they 
did  not  consider  the  developer  unbiased  enough  to  chair  the  Review,  and  (2)  the  recent 
establishment  of  the  new  Command  structure  within  AMC  (including  the  consolidation 
of  ARDEC  and  portions  of  Belvoir  RDEC  as  parts  of  TARDEC)  left  it  as  an  unresolved 
issue  for  some  respondents  at  this  point.  One  possible  reason  for  its  high  usage  at  Natick 
is  that  the  MDA  is  remotely  located  at  ATCOM  in  St.  Louis.  However,  Natick  is  not 
unique  in  being  remotely  located  from  the  MDA.  Both  AATD  and  the  parts  of  TARDEC 
that  were  surveyed  are  also  located  away  from  their  MDAs.  One  unique  organizational 
difference  is  that  the  Natick  MDA  at  ATCOM  is  predominately  concerned  with  aviation 
items.  The  lack  of  direct  involvement  in  the  items  under  development  at  Natick  could 
provide  an  explanation  for  the  high  interest  in  having  the  Review  Chairmanship  at 
Natick.  Interestingly,  the  recent  change  that  places  Natick  under  a  new  MSC,  Soldier 
Support  Command  (SSCOM),  which  is  co-located  at  Natick,  may  change  this  opinion  to 
one  expressed  by  a  respondent  from  ERDEC:  "Like  having  General  Officer  sign  it". 

Technique  18  (Waive  requirement  for  a  separate  Logistics  Demonstration)  is  a 
program  requirement  monitored  and  enforced  by  the  Army  Materiel  Systems  Analysis 
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Activity  (AMSAA).  According  to  the  narrative  comments,  for  non-Natick  commodities, 
the  demonstration  is  not  required  in  many  cases  (for  ammunition  for  instance). 

For  Technique  25  (Utilize  a  single  contract  for  EMD  and  First  Production)  the 
non-Natick  respondents  disagreed  with  the  use  of  the  technique.  Some  of  the  comments 
were: 


Cost  factors  not  fully  known  until  the  EMD  build. 

Should  foster  competition  in  production. 

TDP  should  be  proven  in  production  by  another  source. 

These  comments  express  a  concern  with  selection  of  a  production  contractor  based 
on  cost  estimates  generated  in  Dem/Val  and  the  opinion  that  it  is  in  the  Government’s 
best  interest  to  have  competition  in  the  first  production. 

Overall,  the  techniques  with  the  greatest  differences  in  usage  rates  between  Natick 
and  non-Natick  respondents  were  due  to  disagreements  about  the  value  of  the  techniques 
and  their  inherent  risk  levels,  as  well  as  commodity  and  command  structure  related 
differences. 

Highest  Usage  Comparison  -  To  analyze  Table  1,  it  was  decided  to 
comparatively  analyze  the  highest  usage  rate  techniques  for  the  Natick  and  non-Natick 
groups.  Tables  4  and  5  outline  the  streamlining  techniques  that  received  the  highest 
response  rates.  The  ten  techniques  with  the  highest  Natick  usage  and  the  eleven 
techniques  (there  was  no  distinction  between  usage  rates  of  the  tenth  and  eleventh 
highest)  with  the  highest  non-Natick  usage  are  presented  for  comparison,  to  investigate 
similarities,  differences  and  the  reasons  for  their  high  usage. 

Common  Techniques  -  As  depicted  in  Tables  4  and  5,  of  the  high  ten  or  eleven 
techniques  of  the  two  groups,  six  are  common  to  both  groups.  These  will  be  analyzed 
individually  for  the  reasons  for  their  high  usage  and  commonality. 

The  reasons  for  the  high  usage  of  Technique  1  (Combine  Milestones  0  and  I) 
were  the  low  risk  and  high  technological  maturity  of  AC  AT  IV  programs.  Specific 
comments  included  "Items  always  of  low  technical  risk  and  do  not  warrant  concept 
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ACAT  IV  Streamlining  Techniques 

Frequency  of  Use  by  Command,  Highest  Natick  and  Non-Natick  Responses 

Shading  =  Common  Techniques 


L  Combine  Milestones  0  and  I  {Skip  CE 
Phase) 


2,  Combine  Milestones  I  and  II  (Skip 
Dem/Val  Phase) 


4.  Combine  Dem/Val  development  and 
Operational  Testing  .y-V  y.  . 


11.  Request  Milestone  Decision  Authority 
delegate  Milestone  Decision  Review 
Chairmanship  to  RDEC  Management 


iilitaffjjiilestpne.  incision  Package  prior 
to  receipt  of  Independent 


•114! .  pe  to  Or  Evaluation 

R^msvtoisupporti  a  Milestone  Decision  : 


16.  Waive  requirement  for  a  Cost  and 
Operational  Effectiveness  Analysis  (COEA) 


18.  Waive  requirement  for  a  separate 
Logistics  Demonstration 


•24!,  Utilize  a  single  contract  for  Demi/Val 
and  EMD 


25.  Utilize  a  single  contract  for  EMD  and 
First  Production 


All  Other  Commands 


L:  Combine  Milestones  0  and  I  (Skip  CE 
Phase) 


2.  Combine  Milestones  I  and  2  {Skip  |I| 
Dem/Val  Phase) 


4.  Combine  Dem/Val  Development  and 
Operational  Testing  , 


5.  Conduct  concurrent  Dem/Val 
Development  and  Operational  Testing 


7.  Conduct  concurrent  (overlapping)  EMD 
DT  and  IOT&E 


13.  Staff  Milestone  Decision  Package  prior 
to  receipt  of  Independent 
‘  Evaluadoh/AssfesmehrRepMs;; 


■  14.  Use  Interim  Assessment  dr  Evaluatidn  .. 
Reports  to  support  a  Milestone  Decision  J- 


17.  Waive  requirement  for  a  separate 
System  Threat  Assessment  Report  (STAR) 


24.  Utilize  a  single  contract  for  Dets/Vai ! 
and  EMD 


26.  Conduct  production  First  Article  Tests 
to  validate  development  deficiency 
corrections 


27.  Validate  technical  manuals/support 
system  during  first  production  instead  of 
during  EMD 


Table  4.  Frequency  of  Use  by  Command 


ACAT  IV  Streamlining  Techniques 

Frequency  of  Use  by  Command 

Highest  Natick  and  Other  Commands'  Responses 


Command 


No.  of 

Respondents 


Streamlining 

Technique 


Natick 


Other  Commands 
Total 


#  Yes 


%  Yes 


#  Yes 


%  Yes 


Shading  =  Highest  Usage  for  Each  Group 


Table  5.  Frequency  of  Use,  Highest  Responses 


exploration"  and  "Small  programs  usually  don’t  need  to  prove  out  technology  to  the 
extent  of  more  complex  ones".  Systemic  problems  with  the  development  process  were 
noted  as  driving  the  decision  to  combine  the  Milestones  as  well.  It  was  noted  that 
Milestone  0  cannot  be  conducted  without  an  approved  Mission  Needs  Statement  (MNS) 
and  that  pressures  to  develop  items  quickly  necessitate  initiating  development  prior  to  the 
receipt  of  the  approved  MNS.  The  idea  is  to  obtain  sufficient  progress  by  the  time  the 
MNS  is  approved  to  conduct  the  Milestone  I  Decision  Review.  In  addition,  in  many  low 
tech/low  risk  programs,  the  activities  required  for  Milestones  0  and  I  are  almost 
synonymous. 

The  reasons  cited  for  the  high  usage  of  Technique  2  (Combine  Milestones  I  and 
II)  were  also  risk  related.  The  use  of  a  single  build/test  phase  fits  the  needs  of  most 
AC  AT  IV  programs,  because  they  are  frequently  modified  NDI  efforts.  A  key  factor 
noted  in  the  selection  of  this  technique  was  the  degree  of  success  in  the  Concept 
Exploration  Phase  and  the  demonstration  of  the  required  technologies.  The  technique  is 
used  because  it  saves  a  great  deal  of  funding  and  program  time.  Respondents  from  one 
Command,  TARDEC,  disagreed  with  its  use  citing  that  there  was  no  reason  to  assume 
the  additional  risks  when,  in  their  view,  the  test  costs  were  low.  The  implication  was  that 
Dem/Val  is  necessary  to  avoid  costly  problems  in  EMD.  This  difference  may  be  due  to 
commodity  differences,  or  differing  degrees  of  risk  acceptance. 

Technique  4  (Combine  Dem/Val  DT  and  OT)  was  frequently  used  because  ACAT 
IV  programs  are  relatively  simple  and  have  few  operational  issues.  The  respondents  felt 
that  DT  with  user  troops  was  sufficient  to  identify  and  verify  user  issues.  However,  even 
though  the  technique  had  high  usage,  there  were  difficulties  associated  with 
implementation.  Specific  concerns  were  "Often  difficult  to  get  OEC  and  TEXCOM  to 
agree"  and  "Ability  to  get  concurrence  from  DT  and  OT  testers/evaluators".  An 
additional  programmatic  concern  noted  was  that  Operational  Requirements  Document 
(ORD)  approval  is  required  approximately  one  year  prior  to  the  start  of  OT.  This  could 
create  a  situation  where  DT  is  held  up,  awaiting  the  ORD,  if  a  combined  test  were 
planned. 
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Techniques  13  (Staff  Milestone  Decision  Package  prior  to  receipt  of  Independent 
Evaluation/ Assessment  Reports)  and  14  (Use  Interim  Assessment  or  Evaluation  Reports 
to  support  a  Milestone  Decision)  address  the  same  fundamental  issue  and  the  reasons  for 
their  high  usage  are  the  same.  The  issue  is  that  programs  approaching  a  Milestone 
Decision  have  time-lines  that  require  specific  Review  support  documentation,  and  the 
conduct  of  the  Review  often  falls  in  a  shorter  timeframe  than  the  evaluators/assessors  can 
support  with  fully  staffed  and  coordinated  formal  reports.  Reasons  cited  by  the 
respondents  keyed  on  the  need  to  meet  the  Milestone  date  and  award  follow-on  contracts. 
They  emphasized  that  use  of  these  techniques  presented  little  risk,  since  the  "results  are 
usually  known  long  before  the  final  report".  It  was  interesting  to  note  that  the  narrative 
comments  keyed  on  the  time  saved  and  low  risk,  but  did  not  indicate  that  the  respondents 
felt  that  the  evaluators/assessors  took  too  long.  This  is  surprising  since  the  evaluations 
are  usually  on  the  critical  path  leading  up  to  the  Decision  Review,  and  the  relationship 
between  the  development  managers  and  evaluators  is  typically  adversarial. 

Technique  24  (Utilize  a  single  contract  for  Dem/Val  and  EMD)  was  cited  as  a 
very  large  saver  of  funding  and  time,  while  actually  lowering  the  risks  associated  with 
two  contracts  (award  problems,  transfer  of  knowledge/technologies,  etc.).  One 
respondent  noted: 

Regarding  AC  AT  IV  programs,  the  time  to  complete  Dem/Val  may  be  the 
time  to  award  a  contract.  Purpose  of  separating  is  to  bring  a  complex 
effort  into  manageable  pieces  thereby  reducing  risk.  In  ACAT  IV  we 
usually  have  much  less  complexity  and  are  able  to  manage  the  Dem/Val 
and  EMD  with  less  risk  and  greater  ease. 

This  appears  to  be  an  almost  universally  accepted  strategy  when  the  program 
includes  both  the  Dem/Val  and  EMD  phases.  However,  difficulties  arise  when  trying  to 
write  a  sufficiently  detailed  Scope  of  Work  early  in  Dem/Val,  given  that  specific 
requirements  may  not  yet  be  fully  defined. 
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Unique  Techniques  -  In  the  top  ten/eleven  usage  techniques,  there  were  four  that 
were  unique  to  Natick  and  five  that  were  unique  to  the  other  Commands.  Each  of  these 
will  be  evaluated  for  reasons  for  the  differences. 

Natick  Unique  Techniques  -  Technique  11  (request  MDA  delegate  Milestone 
Decision  Review  Chairmanship  to  RDEC  management)  was  unique  to  Natick.  This 
technique  was  analyzed  earlier  as  having  one  of  the  largest  usage  differences  between 
Natick  and  non-Natick  respondents.  Reasons  for  the  differences  are  cited  previously. 

Technique  16  (Waive  requirement  for  a  COEA)  was  unique  to  Natick,  but  the 
narrative  comments  from  the  other  Commands  also  supported  the  technique.  The  COEA 
was  generally  considered  too  costly  and  inaccurate.  The  only  difference  was  that  a 
number  of  the  non-Natick  respondents  felt  that  the  decision  was  up  to  TRADOC,  and 
they  were  not  authorized  to  decide.  Therefore,  they  did  not  respond  positively. 

Technique  18  (Waive  requirement  for  a  separate  Logistics  Demonstration)  was 
unique  to  Natick.  According  to  the  narrative  comments,  for  non-Natick  commodities,  the 
demonstration  is  not  required  in  many  cases  (for  ammunition  for  instance).  It  appears  to 
be  a  commodity  specific  requirement  at  the  ACAT  IV  level,  dictated  by  the  directions 
from  AMSAA. 

Technique  25  (Utilize  a  single  contract  for  EMD  and  First  Production)  was  unique 
to  Natick.  As  cited  previously,  the  non-Natick  respondents  expressed  concern  with 
selection  of  a  production  contractor  based  on  cost  estimates  generated  in  Dem/Val  and 
they  considered  it  in  the  Government’s  best  interest  to  have  competition  in  the  first 
production.  The  reasons  for  the  difference  may  be  due  to  commodity  differences  relating 
to  item  complexity  and  contractor  past  performance,  and/or  differences  in  organizations’ 
affinity  for  risk. 

Non-Natick  Unique  Techniques  -  Technique  5  (Conduct  concurrent  Dem/Val  DT 
and  OT)  was  unique  to  the  non-Natick  respondents.  This  is  interesting  since  Technique 
4  (Combine  Dem/Val  DT  and  OT)  was  also  one  of  their  highest  usage  techniques  and 
these  two  techniques  are  mutually  exclusive.  A  number  of  non-Natick  respondents 
commented  on  both  techniques  with  the  same  statements.  This  implies  that  they  either 


45 


did  not  recognize  the  difference  or  they  would  consider  both  and  select  the  best  one  for 
the  situation  at  hand.  In  addition  those  respondents  who  stated  that  Technique  4  was  too 
risky  agreed  with  Technique  5.  One  who  disagreed  with  the  use  of  Technique  4  stated 
regarding  Technique  5  that  he  would  "Definitely  support  this,  DT  and  OT  (are)  not 
necessarily  dependent  on  each  other.  Concurrent  testing  will  bring  you  to  completion 
quicker,  shorten  project  schedule  and  labor  dollars".  However,  Natick  respondents 
clearly  preferred  Technique  4  over  Technique  5.  They  indicated  a  preference  for 
combined  testing  over  concurrent  testing  in  Dem/Val.  Non-Natick  respondents  cited  risk 
most  often  as  the  factor  in  their  decision  to  select  concurrency.  Natick  managers  appear 
to  be  more  willing  to  accept  greater  risk  in  the  area  of  test  phasing.  This  difference  in 
philosophy  may  be  due  to  commodity  differences  or  past  experiences  on  the  part  of  either 
group. 

Technique  7  (Conduct  concurrent  EMD  DT  and  OT)  was  unique  to  non-Natick 
respondents  and  this  mirrors  the  results  regarding  Technique  5  above.  Natick  respondents 
preferred  combined  testing  (Techniques  4  and  6)  and  non-Natick  respondents  preferred 
concurrent  testing  (Techniques  5  and  7).  Both  groups  supported  acceleration  of  testing 
in  both  Dem/Val  and  EMD.  But  differences  in  opinion  concerning  acceptable  risk  levels 
leads  to  a  clear  difference  of  opinion  on  how  to  accomplish  this  acceleration. 

Technique  17  (Waive  requirement  for  a  separate  STAR)  was  unique  to  the  eleven 
highest  usage  techniques  of  the  non-Natick  group.  However,  the  Natick  usage  level  was 
also  very  high,  although  not  in  their  top  ten.  In  addition,  it  is  clear  from  the  narrative 
comments  that  there  is  general  agreement  that  the  technique  is  supported  and  that  a 
STAR  for  ACAT  IV  programs  is  generally  not  necessary  and  not  required  of  the 
managers. 

Technique  26  (Conduct  production  First  Article  Tests  to  validate  development 
deficiency  corrections)  was  unique  to  the  eleven  techniques  used  most  often  by  the  non- 
Natick  group.  However,  the  Natick  usage  level  was  also  very  high,  although  not  in  their 
top  ten.  In  addition,  it  is  clear  from  the  narrative  comments  that  there  is  general 
agreement  that  the  technique  is  supported.  Validating  development  deficiency  corrections 
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has  become  one  of  the  purposes  of  First  Article  Testing  (FAT) .  It  makes  good  business 
sense  to  validate  the  corrections  in  First  Article  instead  of  conducting  a  separate 
validation  test  prior  to  a  Milestone  Decision  and  FAT.  Combining  of  testing  needs  is 
being  applied  here  and  as  noted  earlier  with  respect  to  DT  and  OT  in  Dem/Val  and 
EMD. 

Technique  27  (Validate  technical  manuals/support  system  during  first  production 
instead  of  during  EMD)  was  unique  to  non-Natick  respondents.  The  non-Natick 
respondents  indicated  in  their  narrative  comments  that  this  would  be  a  better  test  of  the 
support  system  and  it  avoids  unnecessary  changes  to  the  manuals/support  system  due  to 
configuration  changes  in  EMD.  This  technique  was  not  used  by  Natick,  as  respondents 
indicated  that  waiting  until  production  was  too  risky,  and  that  the  manuals  and  support 
system  were  required  for  evaluation  during  IOT&E.  This  may  be  a  result  of  past  attempts 
by  Natick  managers  to  institute  this  technique  that  were  not  supported  by  other 
acquisition  team  members.  Commodity  differences  may  have  allowed  other  Commands 
to  get  acquisition  team  approval  for  this  technique  when  Natick  had  not  been  able  to. 

Summary  -  In  summary,  the  high  usage  streamlining  techniques  used  by  AC  AT 
IV  managers  that  are  common  to  Natick  and  the  other  AMC  Commands  can  be 
categorized  into  two  groups,  major  program  events  and  programmatic 
reporting/documentation. 

The  common  major  event  techniques  address  combining,  or  concurrency  of 
events.  Managers  combine  Milestones  0  and  I,  or  I  and  II;  they  conduct  combined 
(Natick)  or  conduct  concurrent  (non-Natick)  Dem/Val  and  EMD  DT  and  OT;  they  use 
a  single  contractor  for  Dem/Val  and  EMD;  and  they  utilize  FAT  to  validate  development 
deficiency  corrections.  The  techniques  are  chosen  because  they  save  large  amounts  of 
time  and  funding,  and  they  have  an  acceptable  level  of  risk  for  AC  AT  IV  programs.  The 
common  programmatic  reporting/documentation  techniques  are  the  acceleration  of 
independent  evaluator/assessor  reporting  for  Milestone  Decision  Reviews,  and  the 
waiving  of  the  need  for  a  STAR,  or  COEA  for  their  programs. 


47 


The  high  usage  techniques  that  were  unique  to  either  the  Natick  or  non-Natick 
group  appear  to  be  due  to  risk  acceptability  differences  that  may  be  due  to  commodity 
differences  or  past  experiences.  It  was  noted  that  Natick  managers  were  consistently 
greater  risk-takers  regarding  test  phase  streamlining. 

B.  TECHNIQUE-BY-TECHNIQUE  COST  SAVINGS  AND  SCHEDULE 
REDUCTION  COMPARISONS 

To  evaluate  the  measures  of  effectiveness  associated  with  the  cost  and  schedule 
savings,  this  section  will  begin  with  an  overall  comparison  of  the  cost  savings  of  small 
and  large  programs.  Then  the  techniques  with  the  highest  savings  will  be  evaluated  and 
compared  with  the  techniques  established  in  the  last  section  as  having  the  highest  usage. 
Comparisons  will  then  be  made  between  small  and  large  programs  regarding  the 
differences  in  high  saving  techniques.  Finally,  the  techniques  with  the  lowest  savings  will 
be  evaluated,  as  opposed  to  the  techniques  with  the  highest  usage,  and  then  by  comparing 
small  versus  large  programs.  As  noted  earlier,  the  lowest  saving  techniques  are  being 
evaluated  because  a  number  of  them  were  both  low  in  savings  and  high  in  use,  which  has 
an  interesting  basis. 

1.  Overall  Comparison 

Before  evaluating  the  highest  and  lowest  saving  techniques,  a  review  of  all 
techniques  by  program  sizes  revealed  an  interesting  relationship.  The  relative  technique- 
to-technique  average  cost  savings  established  by  the  respondents  from  small  and  large 
programs,  Figure  5,  is  very  consistent.  Cost  savings  for  small  programs  are  equal  to,  or 
slightly  lower  than,  large  programs  for  almost  every  technique.  A  similar  comparison  of 
the  schedule  reduction  ratings  for  small  versus  large  programs  showed  no  clear  trend. 
Note  that  the  numbers  of  observations  for  schedule  reduction  in  the  small  program 
category  were  small,  with  a  number  of  cells  containing  very  few  or  no  responses  (see 
Table  2). 
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Cost  Savings 
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Figure  5.  Small  vs.  Large  Program  Cost  Savings 


The  cost  saving  comparison  indicates  a  general  agreement  between  small  and  large 
program  managers  as  to  the  relative  cost  saving  potential  of  each  technique.  There  was 
general  agreement  on  which  techniques  saved  the  most  and  which  saved  the  least.  The 
four  techniques  with  the  largest  separations  between  small  and  large  programs  will  be 
analyzed  individually  for  reasons  for  the  large  differences. 

Technique  6  (Combine  EMD  DT  and  IOT&E)  saved  significantly  more  for  large 
programs.  The  reason  is  not  apparent  from  the  narrative  comments  of  the  respondents. 
However,  it  may  be  due  to  the  increased  complexity  of  larger  systems  and  the  associated 
greater  test  costs. 

For  Technique  12  (Request  DA-Directed  Procurement),  the  small  program 
response  rate  (and  associated  projected  cost  savings)  was  zero.  Since  use  of  this 
technique  is  urgency  driven  (and  then  only  as  a  last  resort),  small  programs  may  never 
have  the  level  of  urgency  or  importance  to  warrant  its  use.  This  is  supported  by 
comments  from  small  program  managers  that  included  not  seeing  any  occasion  for  using 
the  technique. 

Technique  17  (Waive  requirement  for  a  separate  STAR)  was  consistently  indicated 
by  small  program  managers,  as  well  as  many  large  program  managers,  as  not  being 
required.  Therefore,  the  benefit  to  them  was  small.  A  few  large  program  managers 
indicated  savings  and  some  of  their  programs  may  be  threat  driven  and  warrant 
significant  effort  with  respect  to  threat  assessment.  One  large  program  manager  noted  he 
supported  the  waiver  concept  "if  the  system  will  be  in  a  limited  or  no  combat 
environment". 

Technique  24  (Utilize  a  single  contract  for  Dem/Val  and  EMD)  had  significantly 
higher  cost  savings  for  large  programs.  Although  the  narrative  comments  did  not  provide 
a  reason  for  the  difference,  it  may  indicate  that  contract  preparation,  evaluation,  and 
execution  cost  savings  for  larger  programs  are  greater  due  to  increased  system 
complexity.  As  the  complexity  of  a  system  increases  the  development  costs  increase,  and 
higher  cost  development  contracts  involve  increased  regulatory  requirements  and  more 
rigorous  proposal  evaluation  procedures,  both  of  which  increase  administrative  costs.  In 
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addition,  consolidation  of  two  complex  contract  requirements  can  save  substantially  by 
eliminating  the  second  contractor’s  start-up  costs.  Also,  a  contractor  for  the  combined 
effort  can  propose  a  lower  cost  program  when  a  longer  duration  effort  is  guaranteed. 
These  points  indicate  that  as  a  program  size  increases,  the  potential  savings  from 
consolidation  of  Dem/Val  and  EMD  contracts  increases  as  well. 

Overall,  there  is  general  agreement  between  small  and  large  programs  regarding 
the  relative  cost  saving  potential  of  each  of  the  streamlining  techniques.  Two  areas  where 
large  programs  deviate  from  the  trend  are  combining  EMD  DT  and  IOT&E,  and  using 
a  single  contract  for  Dem/Val  and  EMD,  both  of  which  may  be  due  to  the  increased 
complexity  of  larger  programs.  Additionally,  large  programs  have  a  greater  need  for,  and 
larger  cost  savings  from,  threat  and  urgency  related  techniques. 

2.  Highest  Saving  Techniques 

The  techniques  with  the  highest  cost  savings  and  schedule  reduction  are 
highlighted  in  Table  6.  The  top  six  techniques  in  each  category  were  selected  for 
evaluation.  By  comparing  the  techniques  with  the  highest  savings  to  those  with  the 
highest  usage,  as  well  as  the  differences  between  small  and  large  programs,  Figure  6, 
a  number  of  interesting  similarities  and  differences  can  be  noted.  A  comparative  analysis 
of  the  techniques  with  high  savings  in  at  least  two  categories  and  those  with  the  highest 
usage  yields  five  techniques  in  both  categories  and  three  that  have  high  savings  but  were 
not  highly  utilized.  The  following  discussion  first  considers  techniques  with  both  high 
savings  and  high  usage,  then  it  addresses  those  with  high  savings  and  low  usage,  and 
finally  it  examines  the  differences  between  small  and  large  programs. 

High  Savings  and  High  Usage  -  Techniques  1  (Combine  Milestones  0  and  I),  2 
(Combine  Milestones  I  and  II),  and  4  (Combine  Dem/Val  DT  and  OT),  from  the  above 
high  usage  discussion  were  noted  by  the  respondents  as  having  been  selected  because 
they  save  large  amounts  of  time  and  funding.  Specific  comments  were: 
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Projected  Savings  by  Program  Size 

Highlighting  Highest  Savings  Shading  =  Highest  Usage  and  Ratings 
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Table  6.  Techniques  with  the  Highest  Savings,  Shading  =  Highest  Usage  and  Highest  Ratings 
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Figure  6.  Projected  Savings  by  Program  Size,  Techniques  with  the  Highest  Savings. 


Technique  1  -  The  total  manpower  costs  and  schedule  associated  with 
doing  a  Milestone  0  will  be  somewhat  costly,  today  we  are  told  that  we 
need  to  deliver  products  in  a  few  years,  doing  an  additional  step  doesn’t 
streamline  the  process. 

Technique  2  -  Shortens  time  and  reduces  number  of  documents. 

Technique  4  -  Cost  of  testing  is  very  expensive,  schedule  can  be  greatly 
shortened,  in  some  situations  the  acquisition  cycle  can  be  cut  in  half. 

The  managers  agree  that,  for  low  risk  programs,  skipping  phases  and  combining 
tests  makes  sense.  For  Technique  1  there  is  a  lack  of  a  need  for  the  event  and  for  the 
other  two  techniques  the  increase  in  risk  is  not  unreasonable  for  low  complexity, 
technologically  mature  programs. 

Technique  18  (Waive  requirement  for  a  separate  Logistics  Demonstration),  a  high 
Natick  usage  technique,  also  had  a  high  payoff  for  small  programs  in  terms  of  both  cost 
and  schedule  savings.  The  need  for  a  logistics  demonstration  was  stated  by  one 
respondent  as  "depending  on  the  complexity  of  the  system".  Therefore,  smaller  programs 
would  have  less  of  a  need  for  a  separate  event,  and  imposing  a  separate  event  would  have 
a  greater  impact  on  their  programs  in  terms  of  percentage  of  the  total  cost  and  schedule. 

Technique  26  (Conduct  production  First  Article  Tests  to  validate  development 
deficiencies),  as  stated  in  the  above  high  usage  analysis,  follows  the  overall  philosophy 
of  combining  events,  especially  tests,  because  combining  saves  the  most  in  terms  of 
program  funding  and  schedule.  Managers  are  fulfilling  the  need  to  validate  the 
corrections  without  having  to  run  a  separate  event.  It  makes  good  business  sense  and  is 
accepted  as  "business-as-usual"  by  AC  AT  IV  managers. 

Although  the  numbers  are  not  statistically  significant,  for  the  above  high  savings 
and  high  usage  techniques  the  ranges  of  cost  and  schedule  savings  noted  by  the 
respondents  were  as  follows: 
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Small  Programs 

Cost  Savings 

Schedule  Reduction 

Technique  1. 

N/A 

1  -  18  MOS. 

Technique  2. 

$0-  >$100K 

1  -  18  MOS. 

Technique  4. 

$100  -  >$100K 

N/A 

Technique  18. 

$0-  >$100K 

1  -  18  MOS. 

Technique  26. 

$10K  -  $100K 

1  -  18  MOS. 

Large  Programs 

Technique  1. 

$10K  -  >$100K 

1  -  >18  MOS. 

Technique  2. 

>$100K 

1  -  >  18  MOS. 

Technique  4. 

$10K  -  >$100K 

N/A 

Technique  26. 

N/A 

1  -  >  18  MOS. 

These  ranges  of  savings 

are  generally  too  broad 

to  be  of  value  to  ACAT  IV 

managers  in  their  streamlining  decision-making.  However,  the  large  program  cost  savings 
ranges  for  Techniques  1  and  2  are  of  value.  The  quantity  of  respondents  answering  each 
was  nine  and  their  responses  were  very  consistent. 

High  Savings  without  High  Usage  -  Technique  3  (Combine  Milestones  I  and  III) 
presents  the  potential  for  the  greatest  savings  since  it  eliminates  most  of  the  development 
process.  But  its  applicability  is  limited  to  NDI  and  modified  NDI  programs  because  the 
technique  allows  for  very  little  test  and  evaluation.  As  stated  by  one  respondent  "Not 
applicable  to  all  systems,  combining  of  the  two  for  an  NDI  makes  sense". 

Technique  6  (Combine  EMD  DT  and  IOT&E)  presents  the  potential  for  savings 
of  the  same  order  as  Technique  4,  which  had  high  savings  and  high  usage.  But  the 
respondents  felt  that  a  combined  test  in  EMD  was  much  more  difficult  and  risky  than  one 
in  Dem/Val.  Comments  included  "Risk  of  failure  is  high  unless  prior  effort  was  very 
successful",  "Often  difficult  to  get  OEC  and  TEXCOM  to  agree",  and  "Too  hard  to 
schedule  user  troops". 
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Technique  10  (Conduct  user  evaluation  in  lieu  of  formal  IOT&E)  was  noted  as 
a  large  saver  of  funding  but  not  time.  Comments  were  that  IOT&E  was  expensive,  but 
that  the  evaluators  and  assessors  would  not  accept  the  results  of  an  informal  user 
evaluation  in  lieu  of  IOT&E.  The  technique  could  save  significant  funding  but  the  lack 
of  tester  control  and  data  collection  capabilities  make  the  informal  evaluation 
unacceptable  in  most  cases. 

In  summary,  the  high  use  of  the  combining  of  milestones  and  testing  phases  by 
ACAT  IV  managers  is  supported  by  the  high  savings  in  program  costs  and  time  that  is 
associated  with  these  streamlining  techniques.  For  small  programs  the  Logistics 
Demonstration  is  less  useful  and  elimination  represents  a  significant  percentage  of  the 
program’s  cost  and  schedule.  Other  techniques  that  could  save  significant  amounts  of 
funding  and  time  are  unique  to  special  cases  such  as  NDI  programs  or  are  impractical 
and/or  risky. 

Small  versus  Large  Programs  Comparison  -  With  respect  to  cost  savings,  four 
techniques  are  common  to  small  and  large  programs.  Techniques  2  and  3,  which  combine 
Milestones  I  and  II,  and  I  and  III,  respectively,  were  high  cost  savers  for  both  size 
programs.  This  is  understandable  given  the  comments  previously  noted,  that  these 
techniques  save  significant  amounts  of  time.  Technique  1  (Combining  of  Milestones  0 
and  I)  was  only  considered  a  significant  cost  saver  for  large  programs.  For  small 
programs  the  concept  exploration  phase  did  not  involve  significant  funding.  For  low 
technology  systems  that  require  no  concept  testing,  but  only  require  a  concept  prove-out 
paper  study,  this  result  can  be  expected. 

In  addition.  Technique  4  (Combining  of  Dem/Val  DT  and  OT)  was  common  to 
both  groups.  This  was  noted  previously  as  a  very  high  saver.  Finally,  Technique  10 
(Conduct  user  evaluation  in  lieu  of  IOT&E)  was  common  to  both  size  programs.  The 
respondents  comments  noted  that  formal  OT,  especially  IOT&E,  was  very  expensive. 

High  cost  saving  techniques  that  were  unique  to  either  small  or  large  programs 
were  Techniques  6,  18,  and  26.  Technique  6  (Combining  of  EMD  DT  and  IOT&E)  was 
unique  to  large  programs  and  the  reason  was  not  discemable  from  the  respondent’s 
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comments.  The  difference  may  be  complexity  related,  where  the  cost  of  IOT&E  increases 
substantially  as  program  complexity  increases.  Technique  18  (Waive  requirement  for  a 
separate  Logistics  Demonstration)  was  unique  to  small  programs.  As  discussed 
previously,  this  technique  saves  a  significant  percentage  of  a  small  program’s  funding  and 
time,  where  low  complexity  warrants  its  use.  Technique  26  (Conduct  production  FAT 
to  validate  development  deficiency  corrections)  was  unique  to  small  programs  but  only 
two  respondents  contributed  to  the  high  score. 

Overall,  the  same  techniques,  the  combining  of  Milestones  and  test  phases,  save 
the  most  funding  for  both  small  and  large  programs.  The  one  exception  was  that  the 
combining  of  Milestones  0  and  I  for  small  programs  did  not  save  significant  dollars, 
possibly  because  the  proof  of  concept  for  low  technology/complexity  systems  requires 
little,  if  any,  breadboard  testing  and  usually  only  requires  a  paper  study. 

With  respect  to  schedule  reduction,  the  results  were  very  similar  to  the  cost 
saving  results.  The  combining  of  Milestones  <0/1, 1/II,  and  I/III)  were  all  high  schedule 
savers  for  both  small  and  large  programs.  The  combining  of  any  Milestones,  even 
Milestones  0  and  I  for  small  programs,  saved  significant  amounts  of  program  time.  The 
combining  of  EMD  DT  and  IOT&E,  Technique  6,  was  also  a  common  high  schedule 
reducer.  But  the  combining  of  Dem/Val  DT  and  OT,  Technique  5,  was  not,  even  though 
it  was  a  high  cost  saver  for  both  groups.  As  was  noted  in  the  previous  section,  the 
complexity  of  testing  in  EMD  and  the  length  of  formal  IOT&E  may  contribute  to  the 
difference. 

Additionally,  Technique  26,  using  FAT  to  validate  development  deficiency 
corrections,  was  common  to  both  groups.  By  utilizing  the  planned  FAT  to  validate  the 
deficiency  corrections,  the  program  can  avoid  a  separate  event  and  can  proceed  out  of 
development,  both  significant  schedule  impacts. 

The  one  technique  that  was  unique  to  small  programs  was  Technique  18  (Waive 
requirement  for  a  separate  Logistics  Demonstration).  See  the  previous  discussions 
concerning  analysis  of  this  point. 


57 


Highest  Saving  Techniques  Summary  -  Overall,  the  greatest  time  and  funding 
savings  can  be  obtained  by  combining  Milestones  and  test  phases.  Concurrency  of  testing 
was  not  one  of  the  highest  savers,  even  though  it  was  noted  as  a  high  usage  technique, 
particularly  for  Commands  other  than  Natick.  Small  and  large  programs  are  very  similar 
with  respect  to  the  highest  saving  techniques,  with  a  few  exceptions.  For  large  programs 
the  expected  range  of  cost  savings  associated  with  Techniques  1  (Combine  Milestones  0 
and  I)  and  2  (Combine  Milestones  I  and  II)  are  greater  than  $10K  and  greater  than 
$100K,  respectively. 

3.  Lowest  Saving  Techniques 

The  techniques  with  the  lowest  cost  savings  and  schedule  reductions  are 
highlighted  in  Table  7.  As  discussed  previously,  the  lowest  saving  techniques  are  being 
evaluated  because  a  number  of  them  were  both  low  in  savings  and  high  in  use.  The 
lowest  six  techniques  in  each  category  were  selected  for  evaluation.  The  amounts  of 
funding  saved  are  too  small  to  be  of  concern  in  the  decision-making  process.  However, 
the  amount  of  time  saved  might  be  of  concern,  if  the  program  were  in  a  time  critical 
phase  (such  as  leading  up  to  a  Milestone  Review  or  contract  award).  By  comparing  the 
techniques  with  the  lowest  savings  to  those  with  the  highest  usage,  and  the  differences 
between  small  and  large  programs,  Figure  7,  a  number  of  interesting  similarities  and 
differences  are  noted. 

Low  Savings  and  High  Usage  -  A  comparative  analysis  of  the  techniques  with 
the  lowest  savings  in  at  least  two  categories  and  those  with  the  highest  usage  yields  five 
techniques  in  both  categories  and  three  that  have  low  savings  and  that  were  not  highly 
utilized. 

Techniques  13  and  14  are  the  two  high  usage  techniques  concerned  with  the 
acceleration  of  Independent  Evaluation/ Assessments  for  Milestone  Decision  Reviews.  As 
discussed  earlier,  the  reason  for  the  high  usage  was  due  to  the  timing  of  the  Milestone 
Decision  Review.  The  time-line  for  a  Review  is  very  tight  and  slippage  is  usually 
considered  unacceptable,  especially  if  follow-on  contract  award  (and  obligation  of 
unprotected  funds)  is  awaiting  the  decision.  With  events  measured  in  days  and  weeks  it 
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Projected  Savings  by  Program  Size 

Highlighting  Lowest  Savings  Shading  =  Highest  Usage  and  Lowest  Ratings 
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Table  7.  Techniques  with  the  Lowest  Savings,  Shading  =  Highest  Usage  and  Lowest  Ratings 


Projected  Savings  by  Program  Size  -  Techniques  with  Lowest  Savings 


Figure  7.  Projected  Savings  by  Program  Size,  Techniques  with  the  Lowest  Savings 


is  understandable  why  the  small  savings  associated  with  the  use  of  these  techniques 
warrants  their  broad  use.  The  high  use  may  indicate  that  these  techniques  become 
necessary  due  to  slippages  leading  up  to  the  Review,  or  delays  by  the 
evaluators/assessors.  But  the  respondent  comments  indicate  that  they  are  utilized  because 
they  make  sense,  they  save  time  at  a  critical  point  in  the  process  and  there  is  little  risk 
with  their  implementation. 

Technique  27  (Validate  technical  manuals/support  system  during  first  production 
instead  of  during  EMD)  also  saved  little,  in  terms  of  schedule,  but  it  was  highly  utilized 
by  Commands  other  than  Natick.  The  cost  savings  associated  with  this  technique  were 
not  high,  eliminating  cost  as  the  reason  for  the  high  usage.  The  non-Natick  respondents 
indicated  that  this  technique  was  a  better  test  of  the  support  system,  whereas  Natick 
respondents  saw  it  as  risky,  or  unrealistic  (has  to  be  done  prior  to  IOT&E).  The  non- 
Natick  respondents  appear  to  favor  the  technique  not  because  of  cost  or  schedule  savings 
considerations,  but  because  it  provides  a  higher  quality  product,  a  factor  outside  of  the 
considerations  of  the  survey. 

Techniques  16  (Waive  COEA)  and  17  (Waive  STAR)  were  also  highly  utilized 
with  low  savings.  As  noted  earlier,  these  techniques  appear  to  be  a  standard  way  of  doing 
business  for  most  AC  AT  IV  programs.  The  documents  are  generally  not  required  and 
they  should  be  considered  for  generation  on  a  "by  exception"  basis  only. 

Low  Savings  without  High  Usage  -  Techniques  that  had  low  savings  and  were 
not  highly  utilized  indicate  the  difficulty  in  their  utilization  and  their  lack  of  usefulness. 
Technique  12  (Request  DA-Directed  procurement  prior  to  Type  Classification-Standard), 
as  discussed  earlier  is  difficult  to  get  approved  and  is  inappropriate  except  for  in  extreme 
emergencies.  Technique  15  (Use  Independent  Evaluator  Briefing  in  lieu  of  written  reports 
to  support  a  Milestone  Decision)  is  shown  to  be  a  less  desirable  option  to  Techniques  13 
and  14,  and  it  accomplishes  the  same  thing  (accelerate  the  evaluation  process). 
Respondents  preferred  some  form  of  written  report  in  advance  of  the  Decision  Review. 
Technique  19  (Utilize  TIWG  as  E3  Requirements  Board)  doesn’t  appear  to  save  program 
time  or  funding.  It  merely  saves  the  need  for  two  staffing  and  coordination  groups. 
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Small  versus  Large  Programs  Comparison  -  With  respect  to  cost  savings,  all 
of  the  lowest  saving  techniques  are  common  to  both  small  and  large  programs.  As  noted 
earlier,  the  amounts  of  funding  saved  are  relatively  meaningless  and  the  narrative 
comments  contain  no  basis  for  this  agreement. 

With  respect  to  schedule  reduction,  Techniques  15  and  19  are  common  to  small 
and  large  programs,  and  as  noted  earlier  they  are  not  highly  utilized.  They  uniformly 
provide  little  savings  for  the  reasons  noted  above.  Technique  27  is  also  common  to  both 
groups  and,  as  noted  earlier,  is  utilized  by  non-Natick  respondents  to  provide  a  better 
product.  Techniques  13  and  14  are  unique  to  large  programs  with  respect  to  their  low 
schedule  savings.  The  ratings  for  small  programs  are  also  fairly  low,  with  averages  that 
are  actually  lower  than  the  large  program  amounts.  Note,  also,  that  the  small  program 
number  of  observations  are  very  low  in  some  cases. 

Lowest  Saving  Techniques  Summary  -  In  general,  small  schedule  reductions  are 
important  when  they  occur  close  to  key  events,  such  as  the  Milestone  Review.  Waivers 
of  documentation,  such  as  the  COEA  and  STAR,  do  not  create  significant  savings,  but 
they  are  generally  not  applicable  to,  nor  required  of,  ACAT  IV  programs.  Some 
techniques,  such  as  validation  of  the  support  system  in  production,  are  valuable  to  some 
respondents  because  of  improved  quality,  not  savings  of  time  or  funding.  Some 
techniques  are  of  little  to  no  value  because  they  either  do  not  save  enough  to  matter,  or 
because  better  alternative  streamlining  methods  exist.  There  are  no  important  differences 
in  savings  between  small  and  large  programs  that  are  discemable  from  the  data. 

C.  COMMANDS’  ACAT  IV  MANAGEMENT  COMPARISON 

As  noted  earlier  when  excluding  MICOM  and  CECOM  from  the  analysis,  the 
management  of  ACAT  IV  level  programs  in  their  commodity  areas  differs  from  that  in 
the  areas  supported  by  the  other  AMC  Commands.  This  difference  will  be  investigated 
by  first  describing  the  differences  between  the  two  management  structures,  then  exploring 
the  reasons  for  the  differences,  and  finally  evaluating  the  pros  and  cons  of  each.  The 
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Appendix  A.  POCs’  comments  and  the  researcher’s  personal  experience  as  a  manager 
at  Natick  form  the  basis  for  these  comparisons. 

The  management  of  small  development  programs  within  the  U.S.  Army  generally 
travels  along  the  following  path:  (1)  the  AMC  RDECs  and  Army  Research  Laboratory 
(ARL)  investigate  technologies  and  consider  concepts  for  development  (Technology  Base 
and  Concept  Exploration  Phases);  (2)  for  the  development  phases,  some  programs 
transition  to  the  Program  Executive  Officer  (PEO)  management  structure,  others  remain 
with  AMC  and  are  managed  by  AMC  level  PMs,  and  still  others  are  left  with  the  AMC 
RDECs;  (3)  following  development  and  fielding,  management  transitions  to  the  AMC 
MSCs  for  sustainment  and  eventual  disposal.  In  the  cases  of  CECOM  and  MICOM,  none 
of  the  programs  remain  in  AMC  when  they  transition  into  development,  they  all 
transition  to  PEO  management. 

Both  of  these  management  philosophies  appear  to  be  allowed  by  DoD  5000,  but 
the  method  employed  for  the  CECOM  and  MICOM  commodities  does  not  appear  to  meet 
the  intent  of  DoD  5000  or  its  basis,  the  Packard  Commission  Findings,  both  of  which 
were  discussed  in  Chapter  II.  The  required  Milestone  Decision  Authority  levels  for  each 
of  the  ACATs  in  DoD  5000  are  listed  in  Figure  2.  In  addition,  the  program  management 
levels  associated  with  each  AC  AT  level  are  defined  in  the  Army’s  implementation 


regulation  for  DoD  5000  (AR  70-1)  and  the  criteria  for  their  selection  are  listed  below: 

ACATs 

Program  Management 

Primary  Criteria 

I 

PEO  PM 

$300M  RDTE,  $1.8B  Procurement 

II 

PEO  PM 

$115M  RDTE,  $540M  Procurement 

III 

PEO  or  AMC  PM 

High  Visibility,  Special  Interest 

IV 

Systems  Manager 

All  Other  Acquisition  Programs 

From  these  criteria,  it  can  be  established  that  only  the  more  important  lower  cost 
programs  should  require  certified  PMs.  In  addition,  the  mission  of  the  PEO  is  stated  as 
the  cost-effective  acquisition  of  major  new  weapon/support  systems.  Whereas,  the 
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related  AMC  missions  are  the  acquisition  of  non-PEO  new  weapon/support  systems  and 
the  improvement  and  sustainment  of  mature,  fielded  systems.  Although  there  does  not 
appear  to  be  a  restriction  against  PEO  management  of  all  development  programs  in  a 
commodity  area,  from  the  above  statements  the  intent  of  the  DoD  and  the  Army  appears 
to  be  that  non-certified  AMC  managers  should  manage  the  development  of  the  smaller 
new  systems  in  each  of  the  commodity  areas. 

There  may  be  some  commodity  related  explanations  for  this  difference.  Programs 
remain  with  Natick  and  ERDEC  because  there  are  no  PEOs  controlling  their 
commodities.  TARDEC  encompasses  ARDEC  munitions  and  support  items  that  have 
carried  over  from  the  disestablished  Belvoir  RDEC  that  do  not  have  related  PEOs.  In 
addition,  AATD  manages  a  number  of  aviation  support  items  that  are  not  controlled  by 
the  PEO  Aviation.  In  the  cases  of  CECOM  and  MICOM,  the  AMC  Commands  and  the 
PEOs  are  directly  aligned,  with  respect  to  commodities,  and  they  are  co-located.  Of  the 
above  three  Commands,  the  only  one  with  a  situation  similar  to  CECOM  and  MICOM 
is  AATD.  This  alignment  and  the  PEOs’  degree  of  control  over  funding  decisions  may 
have  led  to  a  clear  distinction  between  the  missions  of  the  two,  with  an  evolution  to  PEO 
control  of  all  development. 

The  disadvantage  of  the  MICOM/CECOM  model,  according  to  the  MICOM 
POC,  is  that  the  PEOs  have  assumed  control  of  all  development  and  upgrade 
responsibilities,  and  AMC  has  become  completely  focused  on  supporting  the  PEO, 
relinquishing  their  mission  to  improve  mature,  fielded  systems.  The  concern  of  MICOM 
is  that  the  PEOs  do  not  fully  consider  system  upgrades  when  evaluating  the  need  for  a 
new  or  improved  capability.  They  concentrate  on  developing  new  systems. 

The  above  concerns  could  be  labeled  as  an  AMC  Command  complaining  about 
the  PEOs  and  their  own  loss  of  control  over  programs  and  funding.  With  the 
establishment  of  the  PEO  structure  in  the  late  1980s,  the  AMC  Commands  such  as 
CECOM  and  MICOM  lost  a  great  deal  of  funds  control  and  power.  In  addition,  much 
of  AMCs  technical  efforts  transitioned  to  a  role  directly  support  the  PEOs,  and  the  PEOs 
control  of  funding  gives  them  control  over  AMCs  staff.  Both  of  these  factors  have 
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sometimes  lead  to  a  power  struggles  and  attempts  at  "mission-poaching".  However,  the 
fact  that  the  PEOs  control  all  developments  indicates  a  degree  of  over-management  that 
was  not  intended  by  the  Packard  Commission  nor  expressed  in  DoD  5000,  where  ACAT 
III  and  IV  programs  do  not  require  PEO  management.  The  Packard  Commission  findings 
created  the  framework  for  the  Acquisition  Executive  and  PEO  structure  and  they  were 
specifically  directed  at  major  weapon  systems. [Ref.  25]  DoD  5000  allows  for  ACAT  III 
and  IV  management  at  the  lowest  level  deemed  appropriate  by  the  Acquisition  Executive 
(see  Figure  2).  There  is  clearly  a  place  for  management  of  ACAT  IV  size  programs  by 
individuals  who  are  not  certified  PMs,  as  noted  by  the  ACAT  structure  in  DoD  5000  and 
the  allowance  for  AMC  management  of  ACAT  III  and  IV  programs.  In  fact,  this 
management  structure  is  in  place  at  the  other  AMC  Commands  that  participated  in  this 
study. 

On  the  other  hand,  AMC  management  of  existing  programs  and  funding  lines 
suffers  from  a  lack  of  visibility  and  power  when  competing  with  the  PEOs.  The  PEOs, 
if  they  fully  considered  upgrades  when  evaluating  a  new  requirement,  and  if  they 
managed  small  programs  without  creating  new  certified  PMs,  would  be  more  efficient 
as  the  single  point  of  contact  for  all  developments  in  a  given  commodity  area.  In 
addition,  even  the  smallest  programs  would  benefit  from  the  green-suit  management 
available  within  the  PEOs. 

D.  OTHER  COMMENTS 

Two  streamlining  techniques  that  were  proposed  by  the  respondents  to  the  survey 
are  worth  mentioning,  at  least  for  future  reference  or  use  in  follow-on  studies, 

1.  Use  of  capstone  Program  Management  Documentation  such  as  TEMPs,  ORDs. 

Allows  development  of  several  related  efforts  without  waiting  for  separate  staffing 

and  creation  of  multiple  documents. 

2.  Use  of  multi-year  production  contracting  can  result  in  long-term  cost  and  time 

savings. 
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The  first  of  these  techniques  recommends  that  organizations,  rather  than  individual 
AC  AT  IV  managers,  develop  and  maintain  documentation  that  is  required  for  the 
development  of  new  systems  within  their  commodity  areas.  All  systems  requiring 
development  in  a  commodity  area  such  as  heavy  cargo  and  resupply  airdrop  or  a  family 
of  field  heaters,  for  instance,  could  be  supported  with  commodity  level  requirements 
documents  (Acquisition  Strategies,  Test  and  Evaluation  Master  Plans,  etc).  For  low 
complexity  systems  (ACAT  IV),  over-arching  documents  may  unburden  individual 
managers  and,  overall,  save  funding.  Coordination  of  changes  to  the  documents  as  the 
various  efforts  evolve,  however,  could  be  a  burden  and  the  process  could  create  its  own 
bureaucracy.  But  the  concept  is  certainly  worth  considering  for  some  systems  and 
commodities. 

The  second  technique  addresses  the  production  phase  of  materiel  acquisition  which 
this  study  considered  only  when  a  streamlining  technique  involved  a  transition  from 
development  into  production.  If  the  goal  of  development,  and  the  acquisition  process  in 
general,  is  fielding  of  new  and  improved  systems,  production  is  as  important  as 
development.  When  considering  methods  for  cutting  the  cost  and  time  required  to  field 
new  systems,  production  should  be  considered  if  the  entire  process  is  to  be  improved. 
However,  for  many  small  systems,  the  management  of  the  program  transitions  from  the 
RDEC  to  a  procuring  command  prior  to  these  decision.  Analysis  of  both  phases  together 
would  require  investigating  two  distinctly  different  set  of  issues  with  two  completely 
different  groups.  Therefore,  it  is  suggested  that  both  groups  be  investigated,  but  in 
separate,  parallel  efforts. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


The  Primary  Research  Question  for  the  study  was:  "What  acquisition  streamlining 
techniques  are  utilized  by  U.S.  Army  ACAT  IV  managers?"  By  conducting  a  survey  of 
all  of  the  Army’s  ACAT  IV  development  managers,  from  all  of  the  Commands  doing 
ACAT  IV  level  development,  this  thesis  sought  to  ascertain  the  differences  among 
Commands  and  determine  which  techniques  are  most  prevalent,  most  useful,  and  the 
circumstances  that  cause  them  to  be  successful. 

In  support  of  this  thesis,  39  questionnaires  were  distributed  to  the  ACAT  IV 
development  managers  at  various  U.S.  Army  AMC  Commands.  Two  Commands, 
CECOM  and  MICOM,  were  eliminated  from  the  survey  as  is  was  determined  that  all  of 
their  development  programs  were  at  the  ACAT  III  level  or  higher.  A  total  of  19 
responses  were  obtained,  with  some  Commands  providing  limited  responses.  Given  this 
response  level,  a  limitation  was  created  as  it  was  difficult  to  establish  statistical 
significance  for  the  results,  since  the  quantity  of  observations  was  lower  than  expected. 
However,  the  data  that  was  received  was  deemed  to  be  valid.  Therefore,  the  analysis  was 
conducted  on  a  qualitative,  rather  than  predominately  quantitative,  basis.  Following  are 
the  conclusions  and  recommendations  drawn  from  this  analysis,  as  well  as 
recommendations  for  further  research  in  the  subject  area  of  acquisition  streamlining  of 
small  DoD  development  programs. 

A.  CONCLUSIONS 

Subsidiary  Research  Question  1.  -  How  do  Army  Commands  streamline 
ACAT  IV  programs  differently?  The  purpose  of  this  segment  of  the  study  was  to 
determine  if  all  U.S.  Army  Commands  were  aware  of  all  available  streamlining 
techniques  and  establish  if  there  were  differences  in  the  Commands  usage  levels  or 
patterns. 

The  first  area  to  be  analyzed  was  Command  awareness  of  the  techniques.  The 
investigation  found  that  of  the  25  techniques  used  among  the  five  Commands,  there  was 
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only  one  instance  where  none  of  the  respondents  from  a  particular  Command  where 
aware  of  a  specific  technique.  Based  on  this,  it  can  be  concluded  that  cross-fertilization 
of  the  identified  streamlining  concepts  exists  among  the  U.S.  Army  Commands.  The 
information  is  available  to  all  of  them. 

The  second  area  of  analysis  was  a  comparison  of  the  techniques  with  the  greatest 
differences  in  usage  levels  among  the  Commands.  The  relative  usage  levels  for  the 
various  techniques  is  consistent  among  all  Commands.  For  those  techniques  where  the 
usage  differed  the  most,  the  reasons  were  disagreement  with  the  value  or  risk  of  the 
technique,  or  commodity/Command  differences.  The  specific  techniques  with  the  greatest 
differences  were: 

1 .  Combine  Dem/Val  DT  and  OT 

2.  Utilize  a  single  contract  for  EMD  and  First  Production 

3.  Waive  requirement  for  a  separate  Logistics  Demonstration 

4.  Validate  technical  manuals/support  system  during  first  production  instead  of 

during  EMD. 

5.  Request  MDA  delegate  Milestone  Decision  Review  Chairmanship  to  RDEC 

management 

The  first  two  techniques  were  utilized  more  often  by  Natick  managers  because  the 
managers  considered  their  level  of  risk  acceptable.  For  the  third  technique,  managers  in 
the  Natick  commodities  areas  were  more  likely  to  have  this  requirement  imposed  upon 
them.  Therefore,  the  need  for  them  to  consider  a  waiver  was  necessary  more  often.  The 
fourth  technique  was  more  likely  to  be  used  by  managers  at  Commands  other  than 
Natick.  The  managers  at  these  other  Commands  saw  this  technique  as  providing  a  higher 
quality  product.  However,  difficulties  in  getting  acquisition  team  consensus  and  the 
timing  of  events  made  it  less  attractive  to  Natick  managers.  The  need  for  the  fifth 
technique  was  based  on  Command  structure  differences,  where  the  more  remotely  located 
managers  favored  having  the  Chairmanship  delegated. 

The  final  area  of  analysis  for  this  sub-question  addressed  the  techniques  that  were 
utilized  most  often  by  all  of  the  Commands.  A  number  of  streamlining  techniques  are 
used  routinely  by  most  ACAT  IV  programs.  For  techniques  affecting  major  program 
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phases  and  events,  the  risk  levels  associated  with  actions  such  as  the  combining  of 
Milestones  and  the  concurrency  of  test  phases  are  considered  acceptable  for  programs  at 
this  level  of  complexity.  Although  quantifiable  ranges  for  the  levels  of  savings  expected 
for  the  techniques  could  not  be  concluded  from  the  research  (because  of  the  low  number 
of  participants  and  respondents)  the  following  can  be  concluded  as  the  key  techniques 
utilized  by  most  AC  AT  IV  managers. 

1.  Combining  of  Milestones  0  and  I,  and/or  I  and  II 

2.  Combining  of  test  phases,  both  in  Dem/Val  and  EMD,  for  Natick  managers 

3.  Concurrency  of  test  phases,  both  in  Dem/Val  and  EMD,  for  non-Natick 

managers 

4.  Combining  development  deficiency  correction  validation  testing  with  First 

Article  Testing 

5.  Utilize  a  single  contract  for  Dem/Val  and  EMD 

A  difference  was  noted  in  the  test  program  streamlining  philosophies  of  managers 
from  Natick  and  the  other  Commands.  All  Commands  favored  streamlining  through 
concurrent  or  combined  DT  and  OT/IOT&E.  However,  Natick  managers  favored 
combined  testing,  while  managers  from  other  Commands  favored  concurrent  testing,  in 
both  Dem/Val  and  EMD.  Combined  testing  entails  greater  program  risk  and  Natick 
managers  indicated  a  greater  degree  of  risk  acceptance. 

Additionally,  it  was  concluded  that  the  following  techniques  that  address  program 
documents  and  lower-level  actions  were  used  by  most  AC  AT  IV  managers: 

1.  Waive  requirement  for  a  COEA 

2.  Waive  requirement  for  a  separate  STAR 

3.  Staff  Milestone  Decision  Package  prior  to  receipt  of  Independent 

Evaluation/ Assessment  Reports 

4.  Use  Interim  Assessment  or  Evaluation  to  support  a  Milestone  Decision 

It  was  concluded  that  ACAT  IV  programs  generally  do  not  require  COEA  or 
STAR  program  documents.  The  COEA  is  considered  inaccurate,  expensive  and  of  little 
value.  The  STAR  is  generally  unnecessary  because  most  ACAT  IV  programs  are  not 
threat  driven  or  combat  related.  They  are  typically  combat  support  and  combat  service 
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support  systems.  Finally,  independent  evaluation  and  assessment  repotting  for  Milestone 
Decision  Reviews  is  commonly  accelerated  by  staffing  the  Decision  Review  package 
prior  to  receipt  of  an  evaluation/assessment  or  by  utilizing  an  interim 
evaluation/assessment  to  support  the  Decision  Review.  Even  though  these  techniques  do 
not  save  much  program  time  or  funding,  they  represent  low  risk  methods  for  acceleration 
of  the  process  during  the  time-critical  weeks  prior  to  the  Decision  Reviews. 

Subsidiary  Research  Questions  2  and  3  -  How  do  the  various  streamlining 
techniques  compare  with  respect  to  program  cost  savings?  and  How  do  the  various 
streamlining  techniques  compare  with  respect  to  program  schedule  reduction?  The 
purposes  of  these  segments  of  the  study  were  to  consider  cost  and  schedule  savings  to: 
(1)  determine  if  small  and  large  size  program  differences  existed,  and  (2)  investigate 
contributing  factors  to  the  techniques  cited  above  as  having  high  usage.  An  additional 
intended  purpose,  quantification  of  the  expected  cost  and  schedule  savings  for  the 
streamlining  techniques,  was  not  possible  because  of  the  low  number  of  participants  and 
respondents. 

The  comparative  analysis  of  differences  between  small  and  large  size  programs 
considered  the  cost  savings  differences  for  all  techniques,  as  well  as  differences  for  the 
techniques  with  the  highest  and  lowest  cost  and  schedule  reduction  savings.  Large 
programs  were  defined  for  purposes  of  the  analysis  as  those  requiring  $7  million  or  more 
of  RDTE  funding  (for  the  cost  comparisons)  and  three  years  or  more  to  complete 
development  (for  the  schedule  comparisons).  The  analysis  concluded  that  cost  savings  for 
small  ACAT IV  programs  are  equal  to,  or  slightly  lower  than,  large  programs  for  almost 
every  technique.  There  is  close  agreement  concerning  the  relative  cost  savings  associated 
with  each  of  the  streamlining  techniques  in  the  study.  The  two  major  program  areas 
where  large  programs  deviated  from  the  trend  may  be  due  to  the  increased  complexity 
of  larger  programs.  As  program  size  and  complexity  increases,  actions  such  as  IOT&E 
and  contracting  can  grow  dramatically,  with  associated  greater  potential  for  savings 
through  streamlining.  The  two  techniques  were: 
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1.  Combining  EMD  DT  and  IOT&E 

2.  Using  a  single  contract  for  Dem/Val  and  EMD 

In  addition,  it  was  concluded  that  large  programs  had  a  greater  potential  cost 
savings  for  urgency  and  threat  related  techniques.  This  indicates  that  larger  programs 
are  more  likely  to  be  for  combat-related  items,  closer  to  higher  AC  AT  levels,  and  more 
likely  to  require  threat  assessments  and  urgency-related  acceleration  due  to  a  combat 
deficiency  or  enhancement.  The  specific  techniques  were: 

1.  Request  DA-Directed  Procurement 

2.  Waive  requirement  for  a  separate  STAR 

The  areas  of  analysis  that  compared  the  techniques  with  the  highest  usage  and 
those  with  the  highest  and  lowest  savings  supported  the  above  conclusions  for  their  high 
usage.  The  analysis  contributing  additional  insights  into  the  reasons  for  this  high  usage 
and  the  differences  between  small  and  large  programs.  The  frequent  use  of  the  combining 
of  milestones  and  the  concurrency  of  testing  phases  by  ACAT  IV  managers  is  supported 
by  the  high  savings  in  program  costs  and  time  that  is  associated  with  these  streamlining 
techniques.  However,  three  techniques  were  noted  as  potentially  saving  a  great  deal,  but 
they  were  not  used  often.  They  were: 

1.  Combine  Milestones  I  and  III 

2.  Combine  EMD  DT  and  IOT&E 

3.  Conduct  user  evaluation  in  lieu  of  formal  IOT&E  in  EMD 

The  combining  of  Milestones  I  and  III  was  concluded  to  be  of  limited  application, 
to  NDI  or  modified  NDI  programs  only.  The  techniques  of  combining  of  DT  and  IOT&E 
and  the  use  of  a  user  evaluation  in  lieu  of  IOT&E  were  considered  too  risky  and 
extremely  difficult  to  implement,  even  for  programs  of  the  complexity  of  ACAT  I  Vs. 

The  lowest  saving  techniques  were  evaluated  because  a  number  of  them  were  both 
low  in  savings  and  high  in  use.  The  analysis  of  these  techniques  supports  the  reasons  for 
their  high  use.  It  was  concluded  that  techniques,  such  as  the  following,  that  save  small 
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amounts  of  program  time  are  important  when  their  application  supports  key  events,  such 
as  the  Milestone  Review: 

1.  Staff  Milestone  Decision  Package  prior  to  receipt  of  Independent 

Evaluation/ Assessment  Reports 

2.  Use  Interim  Assessment  or  Evaluation  to  support  a  Milestone  Decision 

In  addition,  waivers  of  documentation,  such  as  the  COEA  and  STAR,  do  not 
create  significant  savings,  but  they  are  generally  not  applicable  to,  nor  required  of, 
AC  AT  IV  programs.  Therefore,  it  is  concluded  that  waivers  of  these  requirements  are 
easy  to  obtain. 

Finally,  it  was  concluded  that  some  techniques,  such  as  validation  of  the  support 
system  in  production,  are  valuable  because  they  provide  a  better  product,  not  because  of 
program  funding  or  time  savings.  Some  streamlining  decisions  are  made  based  on  quality 
considerations,  a  factor  not  considered  in  this  study,  where  only  cost  and  schedule  were 
investigated. 

Commands’  ACAT  IV  Management  Comparison  -  The  final  area  of  analysis 
in  this  study  investigated  the  differences  in  the  management  of  ACAT  IV  programs  at  the 
various  AMC  Commands.  As  noted  earlier,  the  analysis  of  streamlining  techniques 
portion  of  this  study  excluded  programs  managed  by  CECOM  and  MICOM,  because  all 
of  their  programs  transitioned  early  in  development  to  PEO  management  (and  became 
ACAT  III  or  higher).  Although  many  programs  at  the  other  Commands  transition  to  PEO 
management  for  development,  there  are  some  programs  in  all  of  the  other  Commands 
that  remain  within  the  management  control  of  the  AMC  Commands,  and  some  of  these 
remain  as  ACAT  IV  (not  requiring  a  certified  PM).  It  was  concluded  that  this  difference 
in  management  control  and  responsibilities  may  be  due  to  commodity  differences. 
Commands  such  as  Natick  and  ERDEC  have  whole  commodity  areas  that  have  no  related 
PEO,  and  all  CECOM  and  MICOM  commodity  areas  are  directly  linked  to  a  PEO.  The 
advent  of  the  PEO  system  allowed  for  a  complete  transfer  of  control  to  the  PEOs  within 
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these  Commands.  This  total  control  of  the  development  programs  in  these  commodity 
areas  by  the  PEOs  may  be  greater  than  was  intended  by  the  Packard  Commission  or  DoD 
5000.  From  the  management  level  selection  criteria  in  the  Army’s  implementation 
regulation  for  DoD  5000  (AR  70-1)  and  the  distinctions  between  the  mission  statements 
of  AMC  and  the  PEOs,  it  was  concluded  that  the  intent  of  DoD  and  the  Army  was  that 
non-certified  AMC  managers  manage  the  development  of  the  smaller  new  systems  in 
each  of  the  commodity  areas.  However,  PEO  management  does  provide  greater  "Green- 
suit"  input  and  this  level  of  control  could  be  more  efficient  if  the  PEOs  did  not  create 
new  organizations  and  certified  PMs  to  manage  these  programs,  and  if  they  considered 
all  avenues  for  meeting  requirements,  including  the  upgrade  of  fielded  systems. 

Summary  -  ACAT  IV  programs  have,  and  can  be  further  streamlined  from  the 
standard  DoD  5000  process  across  programs  and  Commands  in  a  number  of  ways.  The 
following  recommendations  include  changes  that  should  reflect  better  results  than  the 
standard  process,  for  programs  at  this  level  of  complexity. 

B.  RECOMMENDATIONS 

1.  Recommendations  Regarding  ACAT  IV  Management 

It  is  recommended  that  the  results  of  this  study  be  broadly  distributed  to  ACAT 
IV  managers.  Even  though  the  results  provide  no  quantitative  assistance  for  ACAT  IV 
managers  concerning  expected  savings  levels  for  the  various  streamlining  techniques 
investigated,  the  study  does  provide  valuable  information  that  should  assist  managers  in 
their  streamlining  decision-making.  One  respondent  commented,  based  on  seeing  only 
the  survey  form: 

You  bring  up  lots  of  good  ideas.  I  will  keep  a  copy  for  use  the  next  time 

I  prepare  an  Acquisition  Strategy. 

ACAT  IV  managers  can  benefit  from  having  the  list  of  possible  streamlining 
techniques,  from  knowing  which  streamlining  techniques  are  used  most  often,  and  from 
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knowing  the  factors,  such  as  program  risk,  that  they  must  consider  when  developing  a 
streamlining  strategy. 

AC  AT  IV  programs  dramatically  streamline  the  "standard"  DoD  5000  acquisition 
process.  The  process  is  not  the  standard  for  them,  but  for  ACAT  I  programs  and,  as 
such,  the  managers  of  the  ACAT  IV  programs  begin  their  program  planning  process  by 
proposing  significant  changes  from  the  "standard".  This  streamlining  process  requires  the 
concurrence  of  many  acquisition  management  team  members,  some  of  whom  have  no 
incentive  to  support  the  increased  risk  associated  with  streamlining.  This  can  result  in 
program  delays  while  agreements  and  compromises  are  reached  and  significant 
unnecessary  program  costs  and  time  following  portions  of  the  "standard"  process  that  are 
unnecessary  from  a  risk  reduction  perspective.  Therefore,  it  is  recommended  that  the 
U.S.  Army  revise  the  baseline  development  process  for  ACAT  IV  programs,  to  one  that 
incorporates  the  key  streamlining  techniques  of  Combined  Milestones  I  and  II  and 
concurrent  testing  in  EMD  that  were  concluded  in  this  study  as  common  to  most  ACAT 
IV  programs.  A  comparison  of  the  major  phases  and  events  and  their  differences  for  the 
two  "standards"  is  at  Figure  8.  It  is  recommended  that  these  techniques  be  established 
as  standard  procedures  from  which  requirements  are  either  increased  or  decreased.  In 
addition,  it  is  recommended  that  the  alternative  process  exclude  the  requirements  for 
COEA  and  STAR,  except  on  an  "exception  basis"  only.  In  place  of  the  current  system, 
where  the  manager  must  begin  with  the  full  ACAT  I/II  process  and  struggle  with  the 
acquisition  team  members  to  cut  out  elements,  this  recommendation  would  put  risk-averse 
acquisition  team  members  in  a  position  of  justifying  the  need  for  adding  requirements 
versus  the  manager  having  to  justifying  their  exclusion. 

Finally,  it  is  recommended  that  the  Army  evaluate  the  ACAT  IV  management 
structure  that  is  in  place  for  CECOM  and  MICOM  commodities.  There  is  potential  for 
over-management  of  the  small  programs  in  these  commodity  areas  if  they  are  all 
managed  by  certified  PMs.  There  is  a  place  for  non-certified  PM  management  (ACAT 
IV)  in  all  commodities,  as  was  intended  by  DoD  5000.  The  Army  should  consider 
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DoD  5000  Standard  Acquisition  Process 
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Figure  8.  AC  AT  IV  Acquisition  Process  Alternative 


establishing  funding  lines  within  AMC  for  CECOM  and  MICOM  to  manage  the  smallest 
development  programs  in  their  respective  commodity  areas. 

2.  Recommendations  for  Further  Research 

Because  some  of  the  ACAT  IV  managers  contacted  did  not  fully  understand  the 
DoD  5000  process  and  the  standard  terminology  for  acquisition  programs,  it  is 
recommended  that  the  training  needs  of  ACAT  IV  managers  be  investigated.  Note  that 
ACAT  IV  level  managers  are  not  certified  PMs  and,  as  such,  are  not  required  to  be  as 
highly  trained.  However,  because  ACAT  IV  managers  are  typically  civilians  who  do  not 
have  to  rotate  assignments  every  three  years  or  less  (as  ACAT  III  Officers  typically  do), 
they  may  be  fully  effective  as  managers  without  additional  training.  Therefore, 
recommend  a  comparative  analysis  of  the  training  requirements  and  experience  levels  of 
ACAT  III  (certified)  and  ACAT  IV  (non-certified)  managers,  to  determine  whether  the 
experience  and  training  of  non-certified  ACAT  IV  managers  is  adequate. 

Considering  the  lack  of  statistically  significant  results  from  this  study,  it  is 
recommended  that  further  studies  be  conducted  that  concentrate  on  the  most  significant 
techniques,  that  investigate  more  in-depth  the  establishment  of  useful  statistics,  such  as 
confidence  intervals,  for  the  expected  levels  of  cost  and  schedule  savings.  A  major  goal 
of  this  study  that  was  unrealized  was  the  establishment  of  planning  estimates  of  expected 
cost  and  schedule  savings,  for  ACAT  IV  manager  streamlining  decision-making.  This 
type  of  information  would  be  of  great  benefit  to  the  U.S.  Army  ACAT  IV  development 
community. 

Based  on  the  small  number  of  ACAT  IV  managers  in  the  U.S.  Army  and  the 
Command  differences  regarding  CECOM  and  MICOM  (where  all  programs  move  up  to 
at  least  ACAT  III  level),  it  is  recommended  that  future  research  in  this  area  consider  both 
ACAT  III  and  ACAT  IV  managers.  The  purpose  would  be  twofold:  (1)  to  study  the 
differences  between  the  two  groups,  and  (2)  if  they  prove  to  be  similar,  combine  the  two 
for  statistical  information  concerning  expected  savings. 

Based  on  a  recent  briefing  by  the  Commander  of  Marine  Corps  Systems 
Command,  MG  Mutter,  the  Marine  Corps  has  many  more  ACAT  III  and  IV  programs 


76 


than  AC  AT  I  and  II,  and  they  are  actively  streamlining  wherever  possible,  challenging 
all  requirements  of  the  DoD  5000  process.  Their  small  size  and  shorter  chain-of- 
command  provide  for  an  easier  source  of  data  than  was  available  for  this  study  of  Army 
programs  and  may  create  a  streamlining  atmosphere  that  the  Army  could  learn  from. 
Therefore,  it  is  recommended  that  further  research  be  conducted  that  considers  both 
Army  and  Marine  Corps  programs. 

Additionally,  the  following  issues  and  streamlining  techniques  that  were  identified 
during  the  course  of  this  study  are  recommended  for  more  in-depth  investigation: 

1.  The  reasons  cited  most  often  by  ACAT  IV  managers  for  their  streamlining 
decisions;  which  were  risk  level,  and  Command  structure  and  commodity 
differences. 

2.  The  methods  used  for  cross-fertilization  of  streamlining  ideas  among 
Commands. 

3.  Since  quality  was  a  consideration  in  the  selection  of  at  least  one  streamlining 
technique,  validation  of  the  support  system  in  first  production  instead  of  during 
EMD,  further  research  into  how  quality  impacts  the  acquisition  streamlining 
decision-making  process. 

4.  Since  procurement  is  as  important  to  fielding  of  new  systems  as  development, 
it  is  also  recommended  that  ACAT  IV  procurement  streamlining  techniques  be 
investigated  in  a  separate,  parallel  effort. Included  in  this  effort  should  be 
consideration  of  multi-year  production  contracting  as  a  streamlining  technique  that 
can  result  in  long-term  cost  and  time  savings. 

5.  Investigation  of  capstone  Program  Management  Documentation  such  as 
TEMPs,  ORDs  as  a  streamlining  technique.  This  would  allow  for  the 
development  of  several  related  efforts  without  waiting  for  separate  staffing  and 
creation  of  multiple  documents. 
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APPENDIX  A.  ACAT  IV  POINTS  OF  CONTACT 


COMMAND 

NAME 

Natick  RD&E  Center 

Ken  Rice 

Edgewood  RD&E  Center 

Dave  LaBar 

Aviation  Applied 
Technology  Directorate 

Gene  Birocco 

Missile  Command 

Dr.  Gene  Paro 

Communications  & 
Electronics  Command 

Nancy  Roberts 

Tank,  Automotive  and 
Armaments  RD&E  Center 

Jack  Petersen 

Armaments  RD&E  Center 

Bill  Blackwell 

PHONE  NUMBER 
DSN  256-4883 
DSN  584-5272 
DSN  927-2822 

DSN  746-1457 
DSN  992-4159 

DSN  786-6411 

DSN  880-7044 
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APPENDIX  B.  ACAT  IV  MANAGER  QUESTIONNAIRE 
Tailoring  of  the  DoD  5000  Acquisition  Process 

Data  Required  by  the  Privacy  Act  of  1974 


Prescribing  Directive:  AR  70-1 
Authority:  10  USC  4503 

Principle  Purpose:  The  data  collected  by  this  questionnaire  is  to  be  used  for 
research  purposes  only. 

Routine  Uses:  Full  confidentiality  of  the  responses  will  be  maintained  in  the 
processing  of  this  data. 

Mandatory  or  Voluntary  Disclosure  and  Effect  on  Individual  Not  Providing 
Information:  Your  participation  in  this  research  is  strictly  voluntary.  Individuals  are 
encouraged  to  provide  complete  and  accurate  information  in  the  interests  of  the 
research,  but  there  will  be  no  effect  on  individuals  for  not  providing  all  or  any  part  of 
the  information. 


Questionnaire  Instructions 

The  purpose  of  this  questionnaire  is  to  gather  data  on  how  Army  ACAT  IV 
development  managers  tailor  the  DoD  5000  acquisition  process.  You  have  been 
selected  to  receive  this  questionnaire  because  you  have  served  as  an  ACAT 
development  manager  since  the  implementation  of  the  newest  revision  of  the  DoD 
5000  Series. 

-  Please  answer  each  question  on  the  following  pages.  If  you  feel  unqualified 
to  answer  a  particular  question,  leave  it  blank. 

-  Mark  your  answers  directly  on  this  questionnaire  by  placing  an  "X"  over  the 
appropriate  box  or  writing  in  your  response  in  the  space  provided. 

-  When  completed,  please  return  the  entire  questionnaire  in  the  pre-addressed 
government  envelope  provided. 

-  Thank  you. 
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A.  Please  estimate  the  following  ORDERS  OF  MAGNITUDE  totals  for  the  project(s)  that 
you  have  managed  during  the  past  5  years  (Total  per  project  cost  and  duration  estimates  for 
the  period  from  project  initiation  through  the  Milestone  III  decision). 

1.  Development  Cost:  [o-ssooiq  [$sook-$im]  [$im-$3M]  [$3m-s7M]  [S7m-$ism]  [sism-$3im]  i>$3iv 

2.  Development  Duration:  {0-1  Years]  [l-3  Years]  [3-7  Years]  P-15  Years]  [ >  15  Yean 


B.  For  each  of  the  following  potential  acquisition  process  tailoring  methods,  answer  the 
following  three  questions,  followed  by  narrative  comments  concerning  what  positive  or 
negative  factors  lead  you  to  these  selections: 


a.  Have  you  or  would  you 
consider  using  this 
method? 

Yes  No 

[Y]  [N] 


b.  How  would  you  classify 
the  cost  savings  potential 
of  this  method? 


SO-  $100-  S1K-  S10K-  >S100K 

$100  $1000  $10K  100K 

[1]  [2]  [3]  [4]  [5] 


c.  How  would  you  classify 
the  schedule  reduction 
potential  of  this  method? 


0-7  1-4  1-6  6-18  >11 

Davs  Wks  Mos  Mos  Mos 

[1]  [2]  [3]  t4]  [5] 


1.  Combine  Milestones  O  and  I  (Skip  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  f5]  c.  [1]  [2]  [3]  [4] 

Concept  Exploration  Phase) 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 

2.  Combine  Milestones  I  and  II  (Skip  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [3]  [41  '5 

Dem/Va!  Phase) 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 


3.  Combine  Milestones  I  and  III  (Skip  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [3]  [4]  'f 

Dem/Val  &  EMD  Phases) 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 
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4.  Combine  Dem/Val  development  test  (DT)  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [3]  [4]  [5] 
and  operational  test  (OT) 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 

5.  Conduct  concurrent  (overlapping)  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  fl]  [2]  [3]  [4)  [5] 

Dem/Val  DT  and  OT 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 


6.  Combine  EMD  DT  and  Initial  Operational  a.  [Y]  [N]  b.  fl]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [3]  [4]  [5] 
Test  &  Evaluation  (IOT&E) 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 


7.  Conduct  concurrent  EMD  DT  and  IOT&E  a.  fY]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [3]  [4]  [5] 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 

8.  Conduct  Customer  Tests  in  lieu  of  formal  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [3]  [4]  [5] 
DT  in  Dem/Val  Phase 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 

9.  Conduct  Customer  Tests  in  lieu  of  formal  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [31  [4]  [5] 
DT  in  EMD  Phase  '  ‘ 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 
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10.  Conduct  user  evaluation  in  lieu  of  formal  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [3]  [4' 
IOT&E  in  EMD  Phase 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 

11.  Request  Milestone  Decision  Authority  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [3]  K 

delegate  Milestone  Decision  Review 
Chairmanship  to  RDEC  Management 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 

12.  Request  DA-Directed  Procurement  prior  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [3]  [4' 
to  Type  Classification-Standard 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 

13.  Staff  Milestone  Decision  package  prior  to  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [3]  [4] 
receipt  of  Independent  Eval/Assess  Repons 
(IER/IAR),  with  repons  expected  prior  to  the 
decision  meeting. 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 

14.  Use  Interim  Assess  or  Eval  Repons  to  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [3]  [4] 

suppon  a  Milestone  Decision,  with  decision 
contingent  upon  favorable  Final  Repons. 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 
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a.  m  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [3]  [4]  [5] 


15.  Use  an  Independent  Evaluator  briefing  in 
lieu  of  a  report  to  support  a  Milestone 
Decision,  with  written  report(s)  to  follow. 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 
a. 


b. 


c. 

16.  Waive  requirement  for  a  Cost  and  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5] 

Operational  Effectiveness  Analysis  (COEA) 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 

b. 


c. 

17.  Waive  requirement  for  a  separate  System  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5] 
Threat  Assessment  Report  (STAR) 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 

b. 


c. 

18.  Waive  requirement  for  a  separate  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5] 

Logistics  Demonstration 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 
a. 


b. 


c. 

19.  Utilize  TIWG  as  Electromagnetic  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5] 

Environ.  Effects  (E3)  Requirements  Board 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 

b. 


c. 


C.  [1]  [2]  [3]  [4]  [5] 


c.  [1]  [2]  [3]  [4]  [5] 


c.  [1]  [2]  [3]  [4]  [5] 


c.  [1]  [2]  [3]  [4]  [5] 
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20.  Conduct  abbreviated  Logistics  Support  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c. 
Analysis  and  Reporting  (LSA/LSAR) 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 
a. 


b. 


c. 


21.  Don't  prepare  separate  Configuration  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c. 

Management  Plan 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 
a. 


b. 


c. 

22.  Don't  generate  Independent  Life  Cycle  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c. 
Cost  Estimate 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 
a. 


b. 


c. 

23.  Obtain  Milestone  Decision  Authority  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c. 

approval  to  solicit  a  contract  prior  to 
Milestone  Decision  Review 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 
a. 


b. 


c. 


24.  Utilize  a  single  contract  for  Dem/Val  and  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c. 
EMD  Phases 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 
a. 


b. 


c. 


[1]  [2]  [3]  [4]  [5 


[1]  [2]  [3]  [4]  [f 


[1]  [2]  [3]  [4]  [f 


[1]  [2]  [3]  [4]  [5 


[1]  [2]  [3]  [4]  [5 
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25.  Utilize  a  single  contract  for  EMD  and  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [3]  [4]  [5] 

First  Production 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 

26.  Conduct  production  First  Article  Tests  to  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [3]  [4]  [5] 
validate  development  deficiency  corrections 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 

27.  Validate  technical  manuals/support  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [3]  [4]  [5] 

system  during  First  Production  instead  of 
during  EMD 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 

28.  Other  (specify)  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [3]  [4]  [5] 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 

29.  Other  (specify)  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [3]  [4]  [5] 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 

30.  Other  (specify)  a.  [Y]  [N]  b.  [1]  [2]  [3]  [4]  [5]  c.  [1]  [2]  [3]  [4]  [5] 

What  are  the  key  factors  that  lead  you  to  the  above  decisions? 

a. 


b. 


c. 
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APPENDIX  D.  QUESTIONNAIRE  NARRATIVE  COMMENTS  RAW  DATA 


1.  Combine  Milestones  0  and  I  (Skip  Concept  Exploration  (CE)  Phase) 

Respondent  7: 

a.  Items  always  of  low  technical  risk  and  do  not  warrant  concept  exploration 
Respondent  9: 

a.  Small  programs  usually  don’t  need  to  prove  out  technology  to  the  extent  of  more  complex 
ones 

Respondent  10: 

a.  Most  ACAT  IV  efforts  have  NDI  or  NDI-adaptation  solutions 

b.  The  technology  being  considered  is  generally  mature 

c.  Milestone  0  has  very  little  value  added  for  ACAT  IV  efforts 
Respondent  11: 

a.  Maturity  of  technology 

b.  Airdrop  testing  costs 

c.  Availability  of  test  support  (i.e.  aircraft) 

Respondent  13: 

a.  No,  this  is  an  essential  phase  of  R&D 
Respondent  14: 

a.  The  potential  success  of  a  concept  most  of  the  time  can  only  be  proven  out  by  trying  to 
build  a  prototype 

b.  The  total  manpower  costs  and  schedule  associated  with  doing  a  Milestone  0  will  be 
somewhat  costly.  Today  we  are  told  that  we  need  to  deliver  products  in  a  few  years,  doing  an 
additional  step  doesn’t  streamline  the  process 

Respondent  16: 

a.  By  DoD  5000  Milestone  0  occurs  when  the  MNS  is  approved.  Because  of  time  constraints 
almost  all  of  our  programs  are  started  before  the  MNS  is  officially  approved.  Therefore 
technically  we  are  in  Dem/Val  and  our  MNS  is  approved  and  available  for  Milestone  I 
Respondent  17: 

a.  Most  ACAT  IV  items  do  not  require  a  C.E.  phase,  most  issues  can  be  resolved  in  Dem/Val 
or  EMD 
Respondent  21: 

a.  Cost  savings,  schedule  reduction 
Respondent  22: 

a.  Not  much  difference  between  MS  0  and  I.  These  days  if  your  program  is  ready  for  MS  0, 
you’re  probably  ready  for  MS  I 
Respondent  48: 

a.  Basic  technologies  identified,  moderate  risk 

b.  Need  identified  (MNS) 

c.  Type  of  funding  available  (6.3,  6.4,  and  6.5) 

Respondent  49: 

a.  No,  concept  exploration  is  useful  in  determining  system  characteristics/configuration  that 
meets  the  requirement,  I  feel  it  is  needed 

b.  Cost  of  this  phase  will  not  be  reduced,  but  overall  costs  may  be  reduced 

c.  Schedule  in  this  phase  will  not  be  shortened,  but  overall  schedule  may  be 
Respondent  60: 

a.  NDI  approach 

b.  Cost  of  development  bom  by  contractor 

c.  Missing  acquisition  cycle  step 
Respondent  63: 

a.  Complexity  of  contemplated  design  change/improvement 

b.  Evaluate  associated  risk  in  skipping  the  Concept  Exploration  Phase 
Respondent  64: 

a.  Complexity  of  contemplated  requirements,  evaluate  risk  in  skipping  Concept  Exploration 
phase 
Respondent  70: 

a.  Contingent  upon  the  clear,  obvious  demonstration  of  feasible  concepts/technologies  to  be 
applied 

2.  Combine  Milestones  I  and  II  (Skip  Demonstration/Validation  (Dem/Val)  Phase) 

Respondent  3: 

a.  Shortens  time  and  reduces  number  of  documents,  but  allows  for  two  generations  of 
prototype  development  prior  to  MS  III 
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Respondent  7: 

a.  Items  are  typically  of  moderate  technical  risk  and  usually  do  not  warrant  Dem/Val  prior  to 
EMD 
Respondent  9: 

a.  We  are  normally  able  to  start  building  prototypes  of  the  items  we  will  test  early  on 

b.  Funding  for  development  is  always  in  short  supply,  we  usually  have  to  prove  out  our  chosen 
design  while  we  are  building  the  test  items 

Respondent  10: 

a.  Most  ACAT  IV  efforts  have  NDI  or  NDI-adaptation  solutions 

b.  The  technology  being  considered  is  generally  mature,  Milestone  II  is  generally  only  useful 
for  efforts  with  unproven  technology 

Respondent  11: 

a.  Maturity  of  technology,  level  of  success  during  Concept  Exploration 

b.  Airdrop  testing  costs 

c.  Availability  of  test  support  (i.e.  aircraft) 

Respondent  13: 

a.  This  methodology  seems  to  be  taking  place  already 
Respondent  14: 

a.  For  our  programs,  seems  like  its  a  repetition  of  building  a  concept/prototype  and  then  test 

b.  Cost  ana  time  constraints 
Respondent  16: 

a.  NDI  solution  was  not  acceptable  because  after  early  DT  testing  the  ORD  requirements  were 
not  met.  Therefore,  the  program  was  changed  from  I/III  milestone  decision  (one  milestone)  to 
the  next  which  is  I/II  and  III  (two  milestones) 

Respondent  17: 

a.  Most  ACAT  IV  items  do  not  require  a  C.E.  phase,  most  issues  can  be  resolved  in  Dem/Val 
or  EMD 
Respondent  21: 

a.  Cost  savings,  schedule  reduction 
Respondent  22: 

a.  We  did  it 
Respondent  48: 

a.  Basic  technologies  identified,  low  risk 

b.  Need  identified  (MNS) 

c.  Type  of  funding  available  (6.3  and  6.5) 

Respondent  49: 

a.  Off-the-shelf  items  or  even  items  that  have  been  developed  and  are  low  risk  do  not  need  to 
follow  an  extensive  Dem/Val,  I  would  consider  skipping 

b.  The  cost  will  be  reduced  because  you  use  off-the-shelf  items 

c.  The  schedule  will  be  shortened  because  off-the-shelf  items  can  be  procured  quickly 
Respondent  60: 

a.  NDI  approach 

b.  Cost  of  development  bom  by  contractor 

c.  Missing  acquisition  cycle  step 
Respondent  63: 

a.  No,  Dem/Val  is  essential  to  the  development  effort 

b.  Unnecessaiy  to  assume  risk  when  test  costs  are  relatively  small 

c.  Test  costs  following  the  Dem/Val  Phase  escalate  greatly 
Respondent  64: 

a.  No,  no  reason  to  assume  risk  when  test  costs  are  low,  after  Dem/Val  phase  costs  of  tests 
are  high 
Respondent  67: 

a.  We  are  in  fact  using  this  approach.  Procurement  lead  time  savings  are  substantial 
Respondent  70: 

a.  Contingent  upon  the  verifiable,  creditable  demonstration  of  technologies  considered,  must 
take  into  account  the  level  of  maturity  of  given  technology /solution 

3.  Combine  Milestones  I  and  El  (Skip  Dem/Val  and  Engineering  &  Manufacturing  Development 
(EMD)  Phases) 

Respondent  7 : 

a.  Items  are  sometimes  simple  and  pose  little  risk  by  going  directly  to  Milestone  III 
Respondent  9: 

a.  This  is  my  favorite  and  we  are  doing  it  for  the  Modular  General  Purpose  Tent  System. 
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Since  we  normally  only  have  one  shot  at  building  items  we  can  normally  do  a  MS  I/III 

b.  It  saves  test  and  prototype  costs 

c.  It  lets  us  do  2-3  year  programs  vs.  5-7  years,  very  important  to  cut  time  out,  the  user  is 
demanding  quicker  turn-around 

Respondent  10: 

a.  Done  it 

b.  The  maturity  of  most  AC  AT  IV  technology  lends  itself  to  pursuing  a  combined  Milestone 
I/III 

c.  P3I  efforts  can  fix  the  few  issues  left  unresolved  after  Milestone  III 
Respondent  11: 

a.  Maturity  of  technology,  documented  early  successful  testing 

b.  Airdrop  testing  costs  are  high 

c.  Availability  of  test  assets 
Respondent  13: 

a.  No,  by  skipping  the  EMD  Phase,  your  R&D  effort  would  be  left  up  to  the  contractor  once 
you  go  to  production.  The  Army  would  not  know  the  history  of  their  own  programs 
Respondent  14: 

a.  Not  applicable  to  all  systems,  combining  the  two  for  an  NDI  makes  sense.  For  a  non-NDI 
need  to  nave  a  check  point  in  between  Milestones  I  and  II  to  make  sure  all  IPR  members  sign 
on  to  a  concept.  In  light  of  going  away  from  drawings/specs,  on  the  other  hand,  why  have 
extra  milestones  if  you  are  Type  Classifying  a  concept.  Obviously,  eliminating  some  schedule 
will  save  time  but  could  result  in  money  being  put  into  a  dead  end  if  nobody  likes  the  concept. 
Respondent  16: 

a.  This  is  done  for  all  NDI  solutions.  This  is  the  first  thing  we  look  at  for  our  programs, 
because  the  5000  series  requires  NDI  solution  before  any  government  development.  This  also 
allows  the  fastest  and  cheapest  development  which  is  important  for  ACAT  IV  programs 
Respondent  17: 

a.  No,  I  think  you  always  need  to  do  EMD  to  prove-out  technical  issues 
Respondent  22: 

a.  No,  too  much  rushing  of  development 
Respondent  48: 

a.  Basic  technologies  identified,  probably  NDI/modified  NDI 

b.  Need  identified  (MNS  and  ORD) 

c.  Type  of  funding  available 
Respondent  49: 

a. Off-the-shelf  or  low  risk  technology  are  ideally  suited  for  combining  these  phases  because 
little  Dem/Val  or  EMD  effort  is  needed.  I  am  currently  combining  I  and  III  for  an  ACAT  IV 

Erogram 

.  Cost  will  be  reduced  because  of  reduction  in  work 
c.  Schedule  will  be  shortened  also 
Respondent  60: 

a.  No,  disaster  lurks  -  key  stop  is  producibility  and  safety  for  armaments,  must  do  EMD 
Respondent  63: 

a.  No,  Dem/Val  is  essential  to  the  development  effort 

b.  Unnecessary  to  assume  risk  when  test  costs  are  relatively  small 

c.  Test  costs  following  the  Dem/Val  Phase  escalate  greatly 
Respondent  64: 

a.  No,  no  reason  to  assume  risk  when  test  costs  are  low,  after  Dem/Val  phase  costs  of  tests 
are  high 
Respondent  67: 

a.  No,  unacceptable  program  risk  with  this  approach.  Large  number  of  test  quantities  in  EMD 
discourage  elimination  of  EMD  Phase 
Respondent  69: 

a.  Up  to  5  contractors  had  non-developmental  items  potentially  capable  of  meeting 
requirements  in  performance  spec  and  ROC.  Contractors  were  willing  to  provide  3  bid  samples 
for  testing,  at  no  cost  to  the  government,  as  part  of  the  solicitation  process,  leading  to  a 
multiyear  (5  year)  production  contract.  Milestone  III  production  decision  IPR  judged  the 
acceptability  of  the  contractors  test  results  prior  to  award.  Contract  obtaining  both  production 
hardware  and  data  for  transition  to  organic  support  after  2  -  3  years  of  interim  contractor 
support.  Potential  for  cost  savings  was  great  aue  to  competition 
Respondent  70: 

a.  In  support  of  an  NDI  acquisition,  our  team  was  able  to  combine  Milestones  I  and  III  rather 
effectively  with  total  management  support,  this  streamlining  or  tailoring  decision  was  a 
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common  sense  approach  as  the  hardware  was  developed  and  available  as  a  commercial  off-the- 
shelf 


4.  Combine  Dem/Val  Development  Test  (DT)  and  Operational  Test  (OT) 

Respondent  3: 

a.  DT  with  user  troops  is  sufficient  to  identify  and  verify  user  issues 
Respondent  7:  , 

a.  DT/OT  is  a  reasonable  alternative  for  relatively  simple  items  with  few  operational  issues, 
small  emphasis  on  OT 
Respondent  9: 

a.  Time  saver,  cost  saver  on  number  of  prototypes  needed,  testing,  and  reporting 

b.  Often  difficult  to  get  OEC  and  TEXCOM  to  agree 
Respondent  10: 

a.  Testing  on  ACAT  IV  efforts  is  often  overdone 

b.  Proper  engineering  work  can  limit  required  testing 

c.  Many  DT  issues  can  be  looked  at  during  a  thorough  OT 
Respondent  11: 

a.  Time,  maximize  return  from  testing  costs 

b.  Airdrop  testing  costs 

c.  Length  of  an  airdrop  OT 
Respondent  14: 

a.  If  TECOM  does  it,  yes.  TEXCOM,  if  in  the  lead,  is  extremely  costly  and  manpower 
intensive  and  doesn’t  do  well  in  measuring  technical  parameters.  Our  RAM  driven  tests  are 
extremely  similar  to  TEXCOM’s  OT:  Mission  profiles  are  followed.  OT  tests  for  CBPS  cost 
approximately  $700K  versus  one  DT  test  is  approximately  $150K 
Respondent  16: 

a.  Cost 

b.  Time,  because  the  ORD  has  to  be  approved  before  you  can  be  in  the  TSARC  (which  you 
have  to  be  in  a  year  before  OT)  which  is  never  the  case  here 

Respondent  17: 

a.  Makes  a  lot  of  sense,  there  is  often  overlap  between  technical  and  operational  issues, 
separate  tests  stretch-out  the  schedule  too  much 
Respondent  21: 

a.  No,  it  will  be  too  costly  in  case  of  gross  failures 
Respondent  22: 

a.  No,  too  hard  to  schedule  OT  troops 
Respondent  48: 

a.  Degree  of  testing  required 

b.  Ability  to  simultaneously  collect  technical  and  operational  data 

c.  Ability  to  get  concurrence  from  DT  and  OT  testers/evaluators 
Respondent  49: 

a.  Off-the-shelf  items  should  not  need  very  much  DT/OT  testing  because  of  the  low  technology 
risks.  I  have  considered  this  but  it  is  difficult  to  convince  the  testing  folks  that  testing  is  not 
required  or  should  be  limited 

b.  Cost  of  testing  is  very  expensive 

c.  Schedule  can  be  greatly  shortened,  in  some  situations  the  acquisition  cycle  can  be  cut  in  half 
Respondent  60: 

a.  No,  can’t  man-fire  during  Dem/Val  to  support  OT 
Respondent  63:  _  , 

a.  Combining  DT/OT  will  make  testing  more  cost  effective,  i.e.,  save  time  and  funds 

Respondent  64: 

a.  Save  cost,  save  time 
Respondent  67: 

a.  Artillery  ammunition  typically  does  not  require  OT 


Respondent  69:  „  J  , 

a.  Bid  sample  testing  at  APG  demonstrated  system  performance  and  also  conducted 
environmental  testing  which  was  the  equivalent  of  DT  testing.  Combat  Developer  agreed  to 
conduct  an  operational  evaluation  of  the  system  during  bid  sample  testing  so  they  could 
provide  an  independent  assessment  for  the  IPR 

Respondent  70:  .  r .  .  . 

a.  No,  we  as  developers  may  be  biased  when  considering  the  performance/efiectiyeness  or 
systems  under  development.  Truly  recommend  maintaining  an  independent  operational  test  or 
user  test,  customer  test,  etc..  This  provides  developers  a  fresh  perspective  from  customers 
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eyes  (not  just  user  representatives) 

5.  Conduct  concurrent  (overlapping)  Dem/Val  DT  and  OT 
Respondent  7: 

a.  Only  useful  as  a  means  of  accelerating  test  schedule 
Respondent  9: 

a.  OK,  but  probably  not  worth  fighting  for,  this  wouldn’t  save  you  that  much 
Respondent  10: 

a.  Program  dependent  -  easy  to  accomplish  with  smaller  items  and  many  prototypes 
Respondent  11: 

a.  Doesn’t  maximize  return  on  testing  costs,  only  saves  time.  Would  conduct  only  if  directed 
to 

b.  There  would  be  no  savings  because  you  are  duplicating  efforts 

c.  The  concurrent  nature  would  save  time 
Respondent  14: 

a.  No,  if  any  problems  discovered  at  DT  and  you  can’t  correct  them,  you  are  in  trouble  in 
OT.  If  the  system  is  doing  poorly  in  RAM  the  test  will  be  terminated.  OEC  wants  production 
representative  systems 
Respondent  16: 

a.  Try  to  remove  operational  testers  from  the  process,  because  of  cost  and  time.  Also  they 
make  the  AC  AT  IV  program  act  like  a  AC  AT  I,  which  we  can  never  do  because  of  time  and 
money 
Respondent  17: 

a.  Separate  tests  take  too  long 

b.  Overlap  of  DT  and  OT  issues 

c.  May  require  more  test  items  than  are  affordable 
Respondent  22: 

a.  Any  concurrency  is  good 
Respondent  48: 

a.  Degree  of  testing  required 

b.  Ability  to  simultaneously  collect  technical  and  operational  data 

c.  Ability  to  get  concurrence  from  DT  and  OT  testers/evaluators 
Respondent  49: 

a.  One  test  program  should  meet  the  testing  requirements  for  the  program,  this  idea  should  be 
the  way  all  testing  should  be  done,  currently  it  is  almost  impossible  to  conduct  one  test 

b.  Costs  would  be  reduced,  one  test  would  cost  far  less  than  three  separate  tests 

c.  Schedule  would  be  greatly  shortened 
Respondent  60: 

a.  No,  can’t  man-fire  during  Dem/Val  to  support  OT 
Respondent  63: 

a.  Combining  DT/OT  will  make  testing  more  cost  effective,  i.e.,  save  time  and  funds 
Respondent  64: 

a.  Save  cost,  save  time 
Respondent  70: 

a.  Definitely  support  this,  DT/OT  not  necessarily  dependent  on  each  other.  Concurrent  testing 
will  bring  you  to  completion  quicker,  shorten  project  schedule,  save  labor  dollars 

6.  Combine  EMD  DT  and  Initial  Operational  Test  &  Evaluation  (IOT&E) 

Respondent  3: 

a.  DT  with  user  troops  is  sufficient  to  identify  and  verify  user  issues 
Respondent  9: 

a.  Time  saver,  cost  saver  on  number  of  prototypes  needed,  testing,  and  reporting 

b.  Often  difficult  to  get  OEC  and  TEXCOM  to  agree.  Keep  in  mind  that  Natick  does  not 
always  do  separate  Dem.Val,  OT,  EMD,  and  IOT&E  even  now 

Respondent  1 1 : 

a.  Prior  success  rate  of  testing  which  would  reduce  risk  and  increase  confidence  for  success  in 
the  combined  effort 

b.  Airdrop  testing  costs 

c.  Normal  IOT&E  time  allotments  for  airdrop 
Respondent  14: 

a.  ? 

Respondent  16: 

a.  Start  with  NDI  solution  so  most  of  our  testing  is  this  type  DT  combined  with  IOT&E 
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Respondent  17: 

a.  Makes  a  lot  of  sense,  there  is  often  overlap  between  technical  and  operational  issues, 
separate  tests  stretch-out  the  schedule  too  much 
Respondent  21: 

a.  Cost  savings,  schedule  reduction 
Respondent  22: 

a.  No,  too  hard  to  schedule  troops 
Respondent  48: 

a.  Degree  of  testing  required 

b.  Ability  to  simultaneously  collect  technical  and  operational  data 

c.  Ability  to  get  concurrence  from  DT  and  OT  testers/evaluators 
Respondent  49: 

a.  One  test  program  should  meet  the  testing  requirements  for  the  program,  this  idea  should  be 
the  way  all  testing  should  be  done,  currently  it  is  almost  impossible  to  conduct  one  test 

b.  Costs  would  be  reduced,  one  test  would  cost  far  less  than  three  separate  tests 

c.  Schedule  would  be  greatly  shortened 
Respondent  60: 

a.  IOT&E  conducted  as  part  of  EMD,  following  man  rating 
Respondent  64: 
a.  ? 

Respondent  70: 

a.  No,  we  as  developers  may  be  biased  when  considering  the  performance/effectiveness  of 
systems  under  development.  Truly  recommend  maintaining  an  independent  operational  test  or 
user  test,  customer  test,  etc..  This  provides  developers  a  fresh  perspective  from  customers 
eyes  (not  just  user  representatives) 

7.  Conduct  concurrent  (overlapping)  EMD  DT  and  IOT&E 
Respondent  9: 

a.  OK,  but  probably  not  worth  fighting  for.  This  wouldn’t  save  you  that  much 
Respondent  11: 

a.  Only  if  forced  to,  results  in  time  savings  only 

b.  Results  in  no  cost  savings,  because  testing  still  must  be  completed 

c.  The  concurrent  nature  will  save  time 
Respondent  14: 

a.  ? 

Respondent  16: 

a.  No,  if  you  try  to  involve  the  operational  testers  you  will  have  big  problems  with  money, 
time,  and  test  constraints 
Respondent  17: 

a.  Separate  tests  take  too  long 

b.  Overlap  of  DT  and  OT  issues 

c.  May  require  more  test  items  than  are  affordable 
Respondent  21: 

a.  Cost  savings,  schedule  reduction 
Respondent  22: 

a.  Concurrency  is  good  (if  you  have  the  manpower  to  observe  both) 

Respondent  48: 

a.  Degree  of  testing  required 

b.  Ability  to  simultaneously  collect  technical  and  operational  data 

c.  Ability  to  get  concurrence  from  DT  and  OT  testers/evaluators 
Respondent  49: 

a.  One  test  program  should  meet  the  testing  requirements  for  the  program,  this  idea  should  be 
the  way  all  testing  should  be  done,  currently  it  is  almost  impossible  to  conduct  one  test 

b.  Costs  would  be  reduced,  one  test  would  cost  far  less  than  three  separate  tests 

c.  Schedule  would  be  greatly  shortened 
Respondent  60: 

a.  IOT&E  conducted  as  part  of  EMD,  following  man  rating 
Respondent  64: 
a.  ? 

Respondent  70: 

a.  Definitely  support  this,  DT/OT  not  necessarily  dependent  on  each  other.  Concurrent  testing 
will  bring  you  to  completion  quicker,  shorten  project  schedule,  save  labor  dollars 
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8.  Conduct  Customer  Tests  in  lieu  of  a  formal  Development  Test  in  Dem/Val  Phase 

Respondent  9: 

a.  Good  idea,  gets  the  customers  input  up  front,  TIWG  may  be  a  tough  sell  on  this 

b.  Not  a  huge  cost  saver  up  front,  but  could  identify  problem  areas  early  which  could  save 
money 

Respondent  10: 

a.  Definite  advantages  for  very  mature  technologies  and  NDI  efforts 
Respondent  11: 

a.  If  technical  evaluators  allow,  it  provides  flexibility  in  selecting  test  sites,  i.e.  other  than 
TECOM  assets 

b.  Looking  for  lower  cost  test  support 

c.  New  test  site  could  also  save  time  if  private  sector 
Respondent  13: 

a.  No,  R&D  DT  is  very  different  from  Customer  Testing 
Respondent  14: 

a.  No  difference,  cost  and  schedule  the  same.  The  only  thing  to  worry  about  is  that  will 
TECOM  accept  the  results  of  a  customer  test 
Respondent  16: 

a.  Speed,  this  allows  for  a  quick  test,  short  lead  time 

b.  Can  send  prototypes  that  are  not  100%  ready,  also  can  send  technicians  to  support  test, 
have  more  direct  control  of  test 

Respondent  17: 

a.  Good  idea,  but  the  current  philosophy  of  the  testers  and  evaluators  is  that  customer  tests 
don’t  count,  i.e.,  can’t  use  results  for  type  classification  decision 
Respondent  21: 

a.  No,  you  need  to  know  if  system  works  before  giving  it  to  the  troops 
Respondent  48: 

a.  Degree  of  testing  required 

b.  Capabilities  of  Government  test  facilities  vs.  industry  and  cost  differential 
Respondent  49: 

a.  This  method  is  more  customer  oriented  so  you  know  exactly  if  your  product  meets  the  users 
requirements,  the  test  agency  may  not  have  an  appreciation  for  the  critical  issues  and  issues 
that  are  not  that  critical 

b.  Costs  would  be  reduced 

c.  Schedule  would  be  greatly  shortened 
Respondent  60: 

a.  No,  little  advantage,  must  keep  comm,  involved,  leads  to  formal  DT/TT  with  TEMP 
Respondent  63: 

a.  No,  conduct  of  formal  Dem/Val  DT  reduces  risk  of  future  formal  test  efforts 
Respondent  64: 

a.  No,  formal  DT  in  Dem/Val,  reduce  risk 
Respondent  67: 

a.  No,  artillery  requires  safety  tests 
Respondent  70: 

a.  No,  I  considered  operational  testing  same  as  customer  tests,  if  OT  is  not  structured  as  a 
customer  test  then  it  should  be.  In  any  event  conducting  customer  tests  in  place  of  DT  may  not 
allow  us  to  measure  system  effectiveness  without  imposing  lengthy  customer  tests  (i.e.,  years) 

9.  Conduct  Customer  Tests  in  lieu  of  a  formal  Development  Test  in  EMD  Phase 
Respondent  9: 

a.  I  think  we  have  to  get  formal,  structured  testing  sometime 

b.  I  think  customer  testing  this  late  in  the  design  process  might  create  chaos 
Respondent  10: 

a.  Definite  advantages  for  very  mature  technologies  and  NDI  efforts 
Respondent  11: 

a.  If  technical  evaluators  allow,  it  provides  flexibility  in  selecting  test  sites,  i.e.  other  than 
TECOM  assets 

b.  Looking  for  lower  cost  test  support 

c.  New  test  site  could  also  save  time  if  private  sector 
Respondent  14: 

a.  No  difference,  cost  and  schedule  the  same.  The  only  thing  to  worry  about  is  that  will 
TECOM  accept  the  results  of  a  customer  test 
Respondent  16: 
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a.  No,  at  this  stage  of  program  it  is  hard  to  substitute  customer  tests  for  more  formal  DT  by 
TECOM.  Also,  would  get  better  information  from  TECOM  test 
Respondent  17: 

a.  Good  idea,  but  the  current  philosophy  of  the  testers  and  evaluators  is  that  customer  tests 
don’t  count,  i.e.,  can’t  use  results  for  type  classification  decision 
Respondent  22: 

a.  No,  still  need  independent  tester/evaluator 
Respondent  48: 

a.  Degree  of  testing  required 

b.  Capabilities  of  Government  test  facilities  vs.  industry  and  cost  differential 
Respondent  49: 

a.  This  method  is  more  customer  oriented  so  you  know  exactly  if  your  product  meets  the  users 
requirements,  the  test  agency  may  not  have  an  appreciation  for  the  critical  issues  and  issues 
that  are  not  that  critical 

b.  Costs  would  be  reduced 

c.  Schedule  would  be  greatly  shortened 
Respondent  60: 

a.  Little  advantage,  must  keep  comm,  involved,  leads  to  formal  DT/TT  with  TEMP 
Respondent  63: 

a.  No,  conduct  of  formal  Dem/Val  DT  reduces  risk  of  future  formal  test  efforts 
Respondent  64: 

a.  No,  formal  DT  in  Dem/Val,  reduce  risk 
Respondent  67: 

a.  No,  artillery  requires  safety  tests 

Respondent  70:  „  . 

a.  No,  I  considered  operational  testing  same  as  customer  tests,  if  OT  is  not  structured  as  a 
customer  test  then  it  should  be.  In  any  event  conducting  customer  tests  in  place  of  DT  may  not 
allow  us  to  measure  system  effectiveness  without  imposing  lengthy  customer  tests  (i.e.,  years) 

10.  Conduct  user  evaluation  in  lieu  of  formal  IOT&E  in  EMD  Phase 

Respondent  9: 

a.  This  is  the  whole  point  of  IOT&E 

b.  Might  get  better  information  from  real  users 
Respondent  11: 

a.  If  the  operational  evaluators  approve  of  an  alternative  test  site 

b.  Seek  out  lowest  testing  costs 

c.  May  save  time 
Respondent  13: 

a.  Customer  focus,  see  what  the  user  wants/needs  not  the  operational  people  at  OEC 
Respondent  14: 

a.  Formal  IOT&E  are  extremely  expensive,  need  all  logistics  products  (i.e.  LSA  reports, 
RPSTLs,  maintenance  tasks,  training,  TMs)  fully  developed 
Respondent  16:  .  , 

a.  Try  to  conduct  any  testing  that  substitutes  for  formal  OT  because  of  the  operational  testers 
treatment  of  ACAT  IV  programs 
Respondent  17: 

a.  Makes  sense  for  most  Natick  items.  Formal  OT  should  be  for  large,  complex  systems 
Respondent  21: 

a.  Cost  savings 
Respondent  22: 

a.  Have  yet  to  participate  in  well  organized  OT 
Respondent  48: 

a.  Degree  of  testing  required 

b.  Capabilities  of  Government  test  facilities  vs.  industry  and  cost  differential 
Respondent  49: 

a.  This  method  is  more  customer  oriented  so  you  know  exactly  if  your  product  meets  the  users 
requirements.  The  test  agency  may  not  have  an  appreciation  for  the  critical  issues  and  issues 
that  are  not  that  critical 

b.  Costs  would  be  reduced 

c.  Schedule  would  be  greatly  shortened 
Respondent  60: 

a.  No,  need  formal  IOT&E  to  get  OEC  support  for  milestone 
Respondent  64: 
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a.  No,  formal  DT  in  Dem/Val,  reduce  risk 
Respondent  70: 

a.  The  intent  of  any  OT  should  be  user  evaluation.  See  no  cost/time  benefit.  However  will 
allow  for  a  system  truly  reflecting  user  requirements.  We  need  to  focus  and  listen  to  user  to 
meet  their  expectations 

11.  Request  Milestone  Decision  Authority  delegate  Milestone  Decision  Review  Chairmanship  to 
RDEC  Management 

Respondent  3: 

a.  Much  quicker  and  simpler 
Respondent  9: 

a.  Empowerment 

b.  Potential  for  quick  turn-around  is  unlimited 

c.  Type  Classify  when  the  system  is  ready,  not  just  when  the  bureaucracy  is  satisfied  and  has 
spent  all  the  money 

Respondent  10: 

a.  The  MDR  chairmanship  should  be  at  the  lowest  possible  working  level 
Respondent  11: 

a.  Time  savings,  ease  of  coordination  with  an  on-site  activity 

b.  Eliminate  travel  costs 

c.  Reduces  coordination  time 
Respondent  13: 

a.  RDEC  management  will  know  your  program  -  where  an  MDA  is  just  signing  on  the  dotted 
line 

Respondent  14: 

a.  No  advantage  in  cost  and  schedule,  having  it  in  the  RDEC  aids  in  that  the  MDA  can  be 
more  involved  in  the  program  when  in  the  RDEC.  Now  the  MDA  only  gets  involved  in  high 
visibility  programs 
Respondent  16: 

a.  No,  current  change  in  Natick  has  the  MDA  being  established  at  Natick  because  of  the 
starting  of  SSCOM 
Respondent  17: 

a.  Current  MDA  at  ATCOM  is  not  close  enough  to  the  development  program 
Respondent  22: 

a.  No,  like  having  general  officer  sign  it 
Respondent  48: 

a.  No,  see  no  benefit 
Respondent  49: 

a.  No,  the  materiel  developer  knows  the  requirement  for  fielding  items  and  he  is  the  manager 
of  that  item  when  it  is  in  the  field,  RDEC  should  not  be  responsible  for  making  that  decision 
Respondent  60: 

a.  System  being  non-complex 

b.  Schedule  streamlining 

c.  Precedence  set  previously 
Respondent  64: 

a.  ? 

Respondent  67: 

a.  With  new  command,  ARDEC  has  not  yet  got  this  authority 
Respondent  69: 

a.  No,  production  decision  IPR  package  (signed  by  CDR,  ARDEC)  was  assembled  with  test 
reports  and  development  and  operational  independent  assessments  supporting  a  Type 
Classification  Standard  recommendation  and  recommending  award  of  the  multi-year  production 
contract 
Respondent  70: 

a.  This  is  being  done  today,  thus  have  not  implied  time/cost  benefit 

12.  Request  DA-Directed  Procurement  prior  to  Type  Classification-Standard 

Respondent  9: 

a.  No,  confuses  issues 

b.  More  bureaucracy 

c.  I  may  be  missing  some  finer  points  on  this  one 
Respondent  10: 

a.  No,  considered  it  -  actually  a  very  difficult  task,  over  18  different  wickets  to  go  through, 
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not  an  effective  method  under  current  system 
Respondent  11: 

a.  Only  if  the  customer/user  has  an  immediate  need 

b.  Costs  will  still  be  incurred  to  achieve  the  Type  Classification 

c.  Time  savings  to  initial  fielding  will  be  achieved,  not  Type  Classification 
Respondent  14: 

a.  Probably  no  change  in  cost  and  schedule.  With  no  more  specs,  we  could  maybe  combine 
RDTE  into  the  production  phase.  The  advantage  of  having  DA  directed  procurement  is  that 
with  money  in  production  it  helps  keep  funds  in  RDTE  program,  ensures  user  and  IPR 
members  support.  Part  of  our  streamlining  theory  that  we  shall  not  develop  anything  the  Army 
doesn’t  want  to  buy 
Respondent  16: 

a.  No,  have  never  done  this  for  any  of  my  programs  and  can  not  see  any  advantages 
Respondent  17: 

a.  This  was  done  on  SICPS  tent,  needed  to  get  items  fielded  even  though  RDTE  was  not  100% 
complete.  Makes  sense  because  the  soldier  would  rather  have  an  85%  solution  now  than  wait  2 
years  for  the  100%  solution 
Respondent  21: 

a.  Prove  out  technical  data  package 
Respondent  22: 

a.  No,  can’t  see  why  this  would  be  a  good  idea 
Respondent  48: 

a.  No,  see  no  benefit 
Respondent  49: 

a.  No,  no  benefit 
Respondent  60: 

a.  No,  Program  directed  competition 
Respondent  64: 
a.  ? 

Respondent  70:  .  , 

a.  No,  This  is  inconsistent  with  shelving  technology  (i.e. ,  develop  system,  finalize  TDP,  and 
have  available  for  procurement  with  no  POM  forecasts) 

13.  Staff  Milestone  Decision  Package  prior  to  receipt  of  Independent  Evaluation/ Assessment 
Reports  (IER/IAR),  with  reports  expected  prior  to  the  decision  meeting 
Respondent  3: 

a.  This  will  speed  up  program  substantially 
Respondent  9: 

a.  Time  saver! 

b.  Tough  sell  to  TIWG/JWG 
Respondent  10: 

a.  Gets  much  of  the  staff  review  accomplished  without  waiting  for  late  test  reports,  has  worked 
well  in  the  past  for  us 
Respondent  11: 

a.  Level  of  confidence  the  favorable  reports  will  receive 

b.  Travel  costs 

c.  Time  saved  from  holding  an  additional  one  or  two  meetings 
Respondent  13: 

a.  This  would  familiarize  the  MDA  with  your  program 
Respondent  14: 

a.  If  you  have  to  meet  a  Type  Classification  date,  the  evaluators  can  just  present  their 
assessments  at  the  IPR  provided  they  tell  you  their  issues  beforehand  so  there  are  no  surprises. 
If  you  have  the  reports  the  advantage  is  that  maybe  you  can  avoid  the  IPR  and  just  do  it  by 
correspondence 
Respondent  16: 

a.  Time,  need  the  MDS  III  to  be  done  in  time  to  start  the  production  procurement  actions 
(transition  to  ATCOM).  Delays  could  cost  loss  of  procurement  funding 
Respondent  17: 

a.  Final  reports  take  too  long,  results  are  usually  known  long  before  final  report 
Respondent  21: 

a.  Time  savings 
Respondent  22: 

a.  Pretty  risky  -  are  IPR  participants  willing  to  bring  positions  to  meeting  without  final 
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reports? 

Respondent  48: 

a.  Shorten  milestone  decision  process 
Respondent  49: 

a.  I  would  try  this  method  to  expedite  the  Milestone  Decision  Reviews,  although  the  IER/IAR 
are  usually  received  in  a  timely  fashion 

b.  Costs  are  not  significant 

c.  Sometimes  it  could  shorten  the  schedule  significantly 
Respondent  60: 

a.  Smart  business  decision,  low  risk 
Respondent  63: 

a.  Staffing  prior  to  receipt  of  IER/IAR  will  reduce  response  time  for  Type  Classification 
position  statement 
Respondent  64: 

a.  Reduce  response  time 
Respondent  69: 

a.  No,  preliminary  staffmg  was  done  with  preliminary  assessments  but  final  assessments  were 
available  for  formal  Milestone  III  IPR  production  decision 
Respondent  70: 

a.  Manager  should  have  clear  knowledge  of  DT/OT  test  results  and  willing  to  accept 
responsibility  for  IARs  content,  this  action  saves  time 

14.  Use  Interim  Assessment  or  Evaluation  Reports  to  support  a  Milestone  Decision,  with  decision 
contingent  upon  favorable  Final  reports 
Respondent  3: 

a.  This  will  speed  up  program  substantially 
Respondent  9: 

a.  More  time  saved 

b.  Gives  assessor/evaluator  time  to  do  the  reports  right 
Respondent  10: 

a.  Definite  time  saver  with  little  risk 
Respondent  11: 

a.  Provided  the  data  in  the  interim  reports  will  be  favorable  for  the  decision 

b.  Little  to  no  cost  savings 

c.  Time  saved  from  waiting  for  the  final  reports 
Respondent  13: 

a.  This  would  keep  the  decision  process  moving  forward  instead  of  standing  still  waiting  for 
final  reports 
Respondent  14: 

a.  Sounds  risky,  if  the  IPR  signs  up  to  something,  they  should  stick  to  it.  Since  the  final 
decision  isn’t  made  until  final  reports  are  there,  what  is  the  point 
Respondent  16: 

a.  Time,  need  the  MDS  III  to  be  done  in  time  to  start  the  production  procurement  actions 
(transition  to  ATCOM).  Delays  could  cost  loss  of  procurement  funding 
Respondent  17: 

a.  Usually,  not  much  changes  between  the  interim  and  the  final  report 
Respondent  21: 

a.  Cost/time  savings 
Respondent  22: 

a.  Great  idea  if  evaluators  are  willing  to  do  so.  Currently  an  interim  report  is  just  as  hard  to 
get  as  final  so  we  push  for  final 
Respondent  48: 

a.  Shorten  milestone  decision  process 
Respondent  49: 

a.  I  would  try  this  method  to  expedite  the  Milestone  Decision  Reviews,  although  the  IER/IAR 
are  usually  received  in  a  timely  fashion 

b.  Costs  are  not  significant 

c.  Sometimes  it  could  shorten  the  schedule  significantly 
Respondent  60: 

a.  Smart  business  decision,  low  risk 
Respondent  63: 

a.  Use  of  interim  reports/with  contingency  statement  will  reduce  response  times 
Respondent  64: 
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a.  Do  most  of  the  decision  pending  receipt  of  final  report 
Respondent  70:  ...  _  , 

a.  Sounds  reasonable  as  long  as  evaluator  is  willing  to  bless  off  on  an  interim  assessment  that 
conveys  his/her  position 


15.  Use  Independent  Evaluator  Briefing  in  lieu  of  a  reports  to  support  a  Milestone  Decision,  with 
written  report(s)  to  follow 

Respondent  3: 

a.  This  would  be  the  quickest  method 
Respondent  9: 

a.  Common  sense,  MDR  should  believe  evaluator 

b.  More  and  more  time  saved 

c.  Gets  item  to  the  field  quicker 
Respondent  10: 

a.  Hadn’t  considered  -  but  a  good  idea 
Respondent  11: 

a.  Convenience/time 

b.  Shouldn’t  affect  cost 

c.  Save  the  time  that  would  be  lost  waiting  for  the  final  report 

Respondent  14:  _  . 

a.  If  you  have  to  meet  a  Type  Classification  date,  the  evaluators  can  just  present  their 
assessments  at  the  IPR  provided  they  tell  you  their  issues  beforehand  so  there  are  no  surprises. 
If  you  have  the  reports  the  advantage  is  that  maybe  you  can  avoid  the  IPR  and  just  do  it  by 
correspondence.  They  can’t  be  allowed  to  change  their  position  in  the  report  versus  briefing 

Respondent  16:  .  .  ,  .  _  ,  r  . 

a.  They  normally  attend  milestone  decision  and  give  this  briefing  no  matter  the  status  oi  the 

evaluation 


Respondent  17:  , 

a.  The  important  thing  is  to  get  the  information  to  the  IPR  and  MDA,  if  a  briefing  can 
accomplish  this  why  wait  for  the  written  report? 

Respondent  22:  ^  .  _ 

a.  Sounds  simpler,  but  how  do  you  ensure  loop  is  closed  i.e.,  receipt  of  reports? 

Respondent  48: 

a.  Shorten  milestone  decision  process 


Respondent  49: 

a.  No,  I  don’t  see  benefit 
Respondent  60: 

a.  Independent  evaluator  and  key  team  member 
Respondent  63: 

a.  Only  if  the  milestone  decision  requires  a  formal  briefing  that 
correspondence 
Respondent  64: 

a.  If  well  documented 


could  not  be  accomplished  by 


Respondent  70:  ,  ,  .  _  w 

a.  No,  I  consider  an  interim  assessment  for  milestone  decision  but  not  briefing.  Why?  -  May 
circumvent  accountability  -  what  if  independent  evaluator  is  not  available  (sick,  training), 
program  momentum  could  require  a  delegated  representative  or  stand-in  convey  critical 
position  with  decreasing  knowledge  and  accountability 


16.  Waive  requirement  for  a  Cost  and  Operational  Effectiveness  Analysis  (COEA) 
Respondent  3: 

a.  Unnecessary  for  ACAT  IV  programs 
Respondent  9:  _ 

a.  I  don’t  know,  when  would  cost  analysis  be  done  and  by  whom? 

b.  Incumbent  on  the  developer  to  have  figures  developed 

c.  Empowerment 
Respondent  10: 

a.  Very  expensive  and  generally  useless  document 


RespondenrH^^  on  the  procurement  numbers  associated  with  the  effort,  if  the  system  requirement 
is  not  strong,  the  document  may  prove  beneficial  when  selling  the  program 


b.  Cost  to  create  the  document 


c.  Time  saved  from  preparing  the  document 
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Respondent  14: 

a.  Don’t  think  we  do  this  for  ACAT  IVs 
Respondent  16: 

a.  The  cost  for  a  COE  A  is  usually  equal  to  the  total  cost  for  the  whole  RD&E  effort. 
TRADOC  is  deciding  if  all  ACAT  Ills  and  IVs  will  be  excluded  from  the  COEA  requirement 
Respondent  17: 

a.  It  is  impossible  to  be  very  accurate.  This  applies  mostly  to  large,  high  cost  systems 
Respondent  22: 

a.  Haven’t  had  a  COEA  done  that  reflects  realistic  development  item  usage,  COEA  personnel 
unable  to  become  familiar  with  item 
Respondent  48: 

a.  Relative  cost  of  product,  degree  of  user  support 
Respondent  49: 

a.  No,  this  document  is  needed  to  determine  cost  savings  which  is  a  major  part  of  the  decision 
process.  It  also  provides  information  for  budgeting  purposes 
Respondent  60: 

a.  No,  need  to  assure  that  investment  is  smart 
Respondent  63: 

a.  No,  this  should  be  a  TRADOC  decision 
Respondent  64: 

a.  No,  this  is  the  training  school  decision 
Respondent  70: 

a.  Conceptually,  COEAs  are  great.  They  allow  decision  maker  to  justify  their  actions  and 
developmental  efforts.  Unfortunately  the  level  of  effort  involved  for  a  small  program  can 
reduce  any  if  not  all  benefits.  Do  not  see  as  benefit  to  customer  for  small  dollar  programs  - 
only  value  to  manager  for  justifying 


17.  Waive  requirement  for  a  separate  System  Threat  Assessment  Report  (STAR) 
Respondent  9: 
a.  ? 

Respondent  10: 

a.  Most  items  at  Natick  have  no  impact  on  threat 
Respondent  11: 

a.  If  the  system  will  be  in  a  limited  or  no  combat  environment 

b.  Cost  to  create  the  document 

c.  Time  saved  from  preparing  the  document 
Respondent  14: 

a.  Don’t  think  we  do  this  for  ACAT  IVs 


Respondent  16: 

a.  Not  reauired  for  ACAT  IV 


Respondent  22: 

a.  Especially  for  ACAT  IV,  another  big  group  of  people  that  can’t  take  time  to  know  item 
Respondent  48: 

a.  Generally  not  needed  or  applicable  for  aviation  ground  support  equipment 
Respondent  49: 

a.  I  do  not  understand  the  requirement  for  this  report 
Respondent  60: 

a.  Common  sense 
Respondent  64: 
a.  ? 


Respondent  69: 

a.  Determination  made  that  this  was  not  required 


18.  Waive  requirement  for  a  separate  Logistics  Demonstration 
Respondent  3: 

a.  Logistics  issues  can  be  determined  in  DT 

Respondent  9:  . 

a.  We  are  doing  this  now,  it  works.  The  log  folks  have  plenty  of  opportunities  to  input  their 
information  requirements  at  TIWGs 
Respondent  10: 

a.  A  very  expensive  and  generally  unnecessary  event  for  most  ACAT  IV  efforts 
Respondent  11: 

a.  Yes,  if  during  operational  testing  data  was  taken  to  show  usability  and  maintainability  of  the 


111 


product 

b.  Cost  to  set  up  test  and  TDY 

c.  Time  saved  from  setting  up  and  holding  the  log  demo 
Respondent  14: 

a.  Only  if  the  TIWG  concurs,  particularly  OEC  and  schools  that  will  do  maintenance  on  your 
system,  depends  on  complexity  of  system 
Respondent  16: 

a.  Usually  include  any  Log  Demo  requirements  in  the  First  Article  Tests  because  that  is 
usually  the  first  time  the  system  is  produced  according  to  the  spec 
Respondent  17: 

a.  Should  be  done  as  part  of  the  OT  or  DT/OT 
Respondent  48: 

a.  Aviation  ground  support  equipment  is  logistics-related  equipment,  so  Log  Demo  may  be 
redundant  testing 
Respondent  49: 

a.  Logistics  for  off-the-shelf  items  are  limited,  also  the  contractor  can  provide  logistics  support 

b.  Cost  of  a  Logistics  Demonstration  can  be  eliminated 

c.  Schedule  can  be  reduced 
Respondent  60: 

a.  Not  required  for  ammo  items 
Respondent  64: 
a.  ? 

Respondent  69: 

a.  Logistics  Demonstration  was  not  required  during  bid  sample  testing.  The  successful 
contractor  is  providing  complete  logistics  inputs  under  contract  CDRL  requirements 

19.  Utilize  Test  Integration  Working  Group  (TIWG)  as  Electromagnetic  Environmental  Effects 
(E3)  Requirements  Board 

Respondent  9: 
a.  ? 

Respondent  10: 

a.  Not  relevant  to  my  programs 
Respondent  11: 

a.  If  I  can  do  it  saving  time  and  money  I  will,  but  I  currently  have  little  understanding  of  the 
E3  Requirements  Board 
Respondent  14: 

a.  The  E3  Board  doesn’t  make  any  sense,  particularly  for  our  programs 
Respondent  16: 

a.  I  think  this  is  now  a  requirement  for  us 
Respondent  22: 

a.  Need  E3  requirements  spelled  out  clearly  in  DoD  5000  or  other  regulations,  TIWG  must 
become  familiar  with  E3  if  this  is  to  work,  ARDEC  is  unmatched  in  their  knowledge 
Respondent  48: 

a.  We  use  TIWGs  to  manage  all  aspects  of  test  program 
Respondent  49: 

a.  Never  addressed  in  our  programs 
Respondent  60: 

a.  No,  members  of  E3  Board  are  subset  of  TIWG 
Respondent  64: 
a .  ? 

Respondent  69: 

a.  TIWG  and  TEMP  approved  the  EMI  testing  conducted.  EMP  testing  may  be  conducted  for 
information  only  on  First  Article  Test  sample 
Respondent  70: 

a.  TIWGs  define  test  measures  that  allow  us  to  assess  performance  of  a  system  versus  user 
requirement.  In  this  we  ensure  systems  are  safe,  effective,  environmentally  compatible,  do  not 
impose  health  hazards,  survivable,  etc.  No  reason  to  have  E3  separate  from  a  group  of  users, 
independent  evaluators,  logisticians,  etc.  whom  bring  already  independent  views  and  analysis 

20.  Conduct  abbreviated  Logistics  Support  Analysis  and  Reporting  (LSA/LSAR) 

Respondent  9: 

a.  ?  I  don’t  know  enough  about  this 
Respondent  11: 
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a.  Yes,  if  the  system  is  not  too  complex  for  the  user  to  operate  and  maintain 
Respondent  14: 

a.  Can  conduct  abbreviated  reports  only  if  TIWG,  TRR  bodies  agree,  particularly  OEC,  user 
and  maintenance  schools.  If  don’t  have  their  approval,  won’t  get  training  certification  for  OT 
and  the  test  won’t  happen 
Respondent  16: 

a.  If  a  LSA  is  required  it  is  tailored  to  the  program 
Respondent  48: 

a.  We  do  aviation  ground  support  equipment,  not  major  systems.  LSA  is  tailored  accordingly 
Respondent  49: 

a.  Logistics  support  is  very  limited  for  off-the-shelf  items,  contractors  provide  support  also 

b.  Cost  can  be  reduced  appropriately 

c.  Schedule  can  be  shortened  appropriately 
Respondent  60: 

a.  No,  LSAVLSAR  not  required  for  ammo 
Respondent  64: 

a.  ILS  manager  to  make  decision 
Respondent  69: 

a.  No,  no  LSAR  inputs  were  obtained  as  part  of  bid  sample  testing,  but  successful  contractor 
is  doing  full  LSA  as  part  of  contract  CDRL  requirements 
Respondent  70: 

a.  Yes,  this  area  needs  emphasis.  As  the  LSA/LSAR  analysis  and  reporting  was  based  on 
major  systems,  I  have  found  that  mis-proportionate  amounts  of  emphasis  is  placed  on  small 
items  required  little  logistic  support.  Need  common  sense  approaches  -  why  have  five  logistics 
persons  supporting  a  four  member  project  office  team? 

21.  Don’t  prepare  separate  Configuration  Management  Plan 
Respondent  9: 

a.  Not  necessary  for  ACAT  IV,  contractors  are  capable  of  doing  this  without  being  told  to 
Respondent  14: 

a.  Not  sure  what  it  is 
Respondent  16: 

a.  Never  prepared  any  separate  CM  Plan  for  my  program 
Respondent  22: 

a.  A  lot  of  work  to  prepare  and  then  not  follow 
Respondent  48: 

a.  Complexity  and  parts  count  of  system 
Respondent  49: 

a.  Configuration  Management  is  usually  controlled  on  a  business  as  usual  basis,  plans  are  not 
usually  generated 

b.  If  a  plan  was  generated  there  would  be  a  cost  savings 

c.  If  a  plan  was  generated  the  schedule  would  be  shortened 
Respondent  60: 

a.  No,  Configuration  Management  Plan  must  be  specific  to  parties  involved 
Respondent  64: 
a.  ? 

Respondent  69: 

a.  No,  Configuration  Management  Plan  was  not  required  for  bid  sample  testing.  Configuration 
Management  Plan  was  required  from  successful  contractor  as  a  CDRL  requirement 

22.  Don’t  generate  Independent  Life  Cycle  Cost  Estimate 
Respondent  3 : 

a.  No,  this  document  doesn’t  take  long  to  prepare  and  is  very  useful 
Respondent  9: 
a.  ? 

Respondent  14: 

a.  In  the  R&D  cycle,  cost  of  the  system  is  a  guesstimate  and  seems  useless  to  me.  All  the 
numbers  that  come  out  of  it  are  just  even  further  guesses 
Respondent  16: 

a.  Always  prepare  this  LCCE  because  it  is  an  annex  to  the  IPS  and  required  for  all  milestones. 
Also,  since  we  do  not  normally  do  COE  A  this  is  the  major  document  to  analyze  the  cost  of  the 
system 
Respondent  22: 
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a.  No,  program  should  have  schedule  and  program  costs,  how  about  using  that 
Respondent  48: 

a.  Relative  cost  of  product,  degree  of  user  support 
Respondent  49: 

a.  No,  LCCE  are  needed  for  budget  purposes  and  COEAs 
Respondent  60: 

a.  No,  summarized  program 
Respondent  64: 
a.  ? 

Respondent  69: 

a.  Concept  for  fielding  this  NDI  item  was  to  use  interim  contractor  support  while  acquirmg 
provisioning  information  and  technical  data  as  CDRL  items  under  the  production  contract. 
After  interim  contractor  support  period  an  economic  analysis  will  determine  if  transition  to  full 
organic  support  is  in  the  Army's  best  interest 
Respondent  70: 

a.  No,  Fm  not  confident  that  ILLCEs  are  always  effective  or  accurate  (due  to  strict  format 
requirements),  however  realize  their  need  in  addressing  cradle-to-grave  issues.  I  think  this  area 
should  be  more  flexible  in  preparation  to  ensure  effective.  Maybe  more  of  an  assessment  and 
less  concrete  dollar  figures,  i.e.,  what  are  cost  drivers  over  life  cycle  -  discuss  not  estimate 

23.  Obtain  Milestone  Decision  Authority  approval  to  solicit  a  contract  prior  to  Milestone  Decision 
Review 

Respondent  3: 

a.  This  would  definitely  accelerate  program  when  one  is  waiting  for  an  IPR  meeting  before 
awarding  next  phase  of  contract 
Respondent  9: 

a.  Gets  the  item  to  the  field  a  lot  quicker 
Respondent  11: 

a.  Yes,  if  favorable  milestone  decision  is  fairly  positive 

b.  Funding  saved  acting  concurrently 

c.  Time  saved  acting  concurrently 
Respondent  14: 

a.  No,  risky,  a  milestone  will  be  done  upon  completion  of  testing,  doesn't  give  you  the  ability 
to  incorporate  changes  in  contract  unless  by  modification.  Many  contingencies  and  reduction 
can  come  out  of  an  IPR,  in  certain  cases  maybe  a  good  idea,  to  meet  a  schedule 
Respondent  16: 

a.  Have  never  done  this  but  can  see  the  potential  savings  for  time  if  the  program  is  constrained 
by  time 

Respondent  17:  .  .  t  f  ,  j 

a.  Contract  award  would  be  subject  to.  Would  save  time,  especially  on  items  with  long  lead 
times 

Respondent  21: 

a.  No,  too  risky 
Respondent  22: 

a.  Soliciting  okay  but  award  should  wait  until  decision  review 
Respondent  48: 

a.  End  of  fiscal  year  award  with  late  fiscal  year  MDR 
Respondent  49: 

a.  No,  changes  at  the  milestone  decision  review  could  have  contract  implications 
Respondent  60: 

a.  No,  MDA  approval  not  required  to  solicit  for  contract 
Respondent  64: 
a.  ? 

Respondent  69: 

a.  No,  Milestone  Decision  Authority  had  test  reports  and  independent  assessments  from  all 
contractors  bid  samples  tested  before  recommending  production  contract  award  and  Type 
Classification  Standard 

24.  Utilize  a  single  contract  for  Dem/Val  and  EMD 
Respondent  3: 

a.  This  is  a  must  in  order  to  have  any  type  of  acceleration 
Respondent  7: 

a.  Lower  risks  -  corporate  history /knowledge  of  contractor  helps  reduce  risks  (and  schedule)  in 
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second  iteration 
Respondent  9: 

a.  We  do  this  now,  basically,  it  works  for  us 

b.  Who  has  time  (18+  months)  to  wait  for  procurement  to  award  another  contract 

c.  Money  sits  unused  while  procurement  and  legal  diddle  with  the  contract 
Respondent  11: 

a.  Yes,  but  I  would  have  to  have  a  huge  level  of  confidence  in  the  system  and  that  the 
Milestone  II  decision  would  be  favorable.  If  I  was  going  to  hold  a  Milestone  I/III,  no  problem 

b.  Funding  saved  to  generate  the  new  contract 

c.  Time  saved  to  generate  the  new  contract 
Respondent  14: 

a.  Saves  time  of  preparing,  soliciting,  and  awarding  contract.  If  new  contractor,  there  will  be  a 
learning  curve,  starting  a  new  contract  will  probably  cost  more  than  continuing  with  the 
previous  one 
Respondent  16: 

a.  Normally  done  because  of  our  NDI  approach  and  the  combining  of  Milestones  I/III 
Respondent  17: 

a.  Saves  time  by  not  re-soliciting,  better  to  have  same  contractor  -  no  learning  curve  in  EMD 
phase 
Respondent  21: 

a.  Eliminates  preparation  of  follow-up  contract,  saves  time/money,  better  learning  curve  effect 
Respondent  48: 

a.  Technical  risk  well  understood  and  controllable,  availability  of  funds 
Respondent  49: 

a. This  approach  would  allow  you  to  develop  the  final  configuration  during  Dem/Val  and 
smoothly  transition  to  EMD 

b.  Cost  for  one  contract  award  can  be  eliminated 

c.  Schedule  would  be  shortened  because  of  one  less  contract  award 
Respondent  60: 

a.  Phases  combined 
Respondent  64: 

a.  Save  cost,  save  time 
Respondent  67: 

a.  We  are  utilizing  this 
Respondent  69: 

a.  No,  contractors  provided  their  systems  for  bid  sample  testing  and  supported  them  through 
the  test  at  no  cost  to  the  Government.  The  production  contract  award  was  sufficient  incentive 
for  their  participation 
Respondent  70: 

a.  Regarding  ACAT  IV  programs,  the  time  to  complete  Dem/Val  may  be  the  time  to  award  a 
contract.  Purpose  of  separating  is  to  bring  a  complex  effort  into  manageable  pieces  thereby 
reducing  risk.  In  ACAT  IV  we  usually  have  much  less  complexity  and  are  able  to  manage  the 
Dem/Val  and  EMD  with  less  risk  and  greater  ease 

25.  Utilize  a  single  contract  for  EMD  and  First  Production 
Respondent  3: 

a.  It  would  be  great  to  have  a  single  contract  to  include  Dem/Val,  EMD,  and  Production 

b.  It  would  reduce  time  and  add  consistency 

c.  This  would  be  a  great  incentive  to  the  contractor  as  well 
Respondent  7: 

a.  Have  not  done  -  but  expect  this  to  become  the  standard  operating  procedure  in  the  future 
because  of  the  new  rules  against  using  military  specs  and  technical  data  packages 
Respondent  9: 

a.  Great  idea,  lets  me  fix  engineering  problems  before  they  hit  high  volume  production 
Respondent  11: 

a.  Yes,  provided  the  first  production  was  a  contract  option 

b.  Save  the  funding  associated  with  a  new  solicitation 

c.  Save  time  to  prepare  the  solicitation 
Respondent  14: 

a.  Wastes  time 

b.  Costs  for  start-up  with  each  new  contract 
Respondent  16: 

a.  Have  never  done 
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Respondent  17:  . 

a.  Expertise  gained  by  contractor  during  EMD  phase  would  be  valuable  in  production,  could 
use  a  less-than-100%  technical  data  package.  Shorter  lead  time  to  production,  tooling  and 
ordering  materials  could  be  done  sooner 
Respondent  21: 

a.  Eliminates  follow-up  contractor,  can  hold  contractor  responsible  for  meeting  design-to-cost 
goals,  assess  production  readiness  better 
Respondent  48: 

a.  Technical  risk  low  and  well  understood 

b.  Strong  user  support  and  need 

c.  Funds  available 

Respondent  49:  ,  .  ..  , , , 

a.  With  a  low  risk  program  that  will  not  see  many  changes  during  EMD,  this  would  be 
appropriate 

b.  Cost  for  one  contract  award  can  be  eliminated 

c.  Schedule  would  be  shortened  because  of  one  less  contract  award 
Respondent  60: 

a.  No,  cost  factors  not  fully  known  until  after  EMD  build 
Respondent  63:  ....  , 

a.  No,  development  and  production  should  be  separate  entities  -  production  contract  may  be 
competed  to  reduce  item  cost 
Respondent  64: 

a.  No,  should  foster  competition  in  production  contract 
Respondent  69: 

a.  No,  went  from  bid  sample  testing  directly  to  a  production  contract  award 
Respondent  70: 

a.  No,  I  do  not  necessarily  agree  with  concept  of  having  first  production  by  contractor 
developing  system  for  small  programs.  I  think  TDP  should  be  proven  in  production  by  another 
source  to  work  out  growing  pains.  Many  problems  in  TDP  may  not  come  out  if  given  to  same 
contractor  -  just  put  off  to  later  date.  Idea  ok  if  awarding  two  production  contracts  to  two 
contractors 


26.  Conduct  production  First  Article  Tests  to  validate  development  deficiency  corrections 

Respondent  9:  , 

a.  We  do  this  now,  good  opportunity  for  everyone  to  get  many  problems  out  m  the  open 

Respondent  11: 

a.  I  believe  the  validation  of  deficiency  corrections  should  be  done  before  the  First  Article 
Test,  although  if  the  relationship  with  the  contractor  is  strong  it  may  be  worthwhile 

b.  Could  save  test  costs 

c.  Could  save  up  to  6  months 
Respondent  14: 

a.  Objective  of  First  Article  Tests 

Respondent  16:  ,  ____  w  ,  .  .  . 

a.  This  is  done  with  the  recommendations  made  by  the  evaluators.  TECOM  has  the  materiel 
release  authority  so  they  normally  check  the  FAT  to  see  if  the  corrections  have  been  made 
before  they  will  do  the  materiel  release 
Respondent  17: 

a.  Allows  you  to  skip  one  of  the  development  phases.  This  was  done  on  SICPS  Rigid  Wall 
Shelter,  good  way  to  prove-out  minor  corrections  to  problems  in  DT 
Respondent  21 : 

a.  Cost  savings,  schedule  savings 
Respondent  22: 

a.  Want  to  check  production  ASAP 
Respondent  48: 

a.  Reduce  scope  and  timeline  requirements  of  Government  DT/OT  testing 
Respondent  49:  . 

a.First  Article  Tests  are  usually  conducted  on  a  business  as  usual  basis 
Respondent  60: 

a.  First  Article  Test  validates  final  product 


Respondent  63:  . 

a.  No,  development  and  production  should  be  separate  entities  -  production  contract  may  be 
competed  to  reduce  item  cost 
Respondent  64: 
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a.  No,  should  foster  competition  in  production  contract 
Respondent  69: 

a.  Conducting  weapon  firing  tests  (Government  test)  as  part  of  Initial  Production/First  Article 
Tests  to  further  demonstrate  system  reliability  which  was  not  adequately  demonstrated  in  bid 
sample  testing 
Respondent  70: 

a.  No,  This  is  commonly  done  and  unfortunately,  with  less  than  desirable  results.  Leads  to 
band-aid  fixes  that  may  never  be  truly  fixed  and  a  dissatisfied  customer 

27.  Validate  technical  manuals/support  system  during  first  production  instead  of  during  EMD 
Respondent  3: 

a.  Accelerates  Type  Classification 
Respondent  9: 

a.  Good  because  engineering  changes  cause  manual  changes  which  cost  money 

b.  You  don't  waste  time  on  the  manuals  during  EMD 
Respondent  11: 

a.  No,  I  believe  it  would  be  risky  to  evaluate  the  manuals  after  EMD  especially  in  todays 
environment  where  contractors  manuals  will  be  used  as  the  military  manuals 
Respondent  13: 

a.  No,  this  would  probably  take  too  much  time  during  production 
Respondent  14: 

a.  Has  to  be  done  prior  to  IOT&E 
Respondent  16: 

a.  ATCOM  is  responsible  for  our  Technical  Manuals  and  they  are  done  in  the  production 
contract.  Most  of  our  designs  are  built  for  the  first  time  in  the  production  contract  (new 
contractor,  performance  spec,  etc.) 

Respondent  17: 

a.  Usually,  there  is  not  enough  RDTE  funding  to  buy  manuals  during  EMD.  If  you  make 
design  changes  during  First  Production,  the  manuals  will  have  to  change  anyway 
Respondent  21: 

a.  If  EMD  and  first  production  are  performed  under  same  contract  or  by  same  contractor 
Respondent  22: 

a.  Better  test  of  how  support  system  will  work 
Respondent  48: 

a.  Reduce  EMD  timeline  by  deferring  some  if  not  all  logistics  activities  to  production 

b.  Risk  of  deferment 
Respondent  49: 

a. This  will  ensure  the  technical  manuals  are  applicable  for  the  production  units 
Respondent  60: 

a.  Easy  to  do  in  EMD 
Respondent  63: 

a.  The  TDP  is  firmer/more  mature  during  first  production  and  avoids  unnecessary  change  if 
conducted  during  EMD 
Respondent  64: 

a.  Cost  of  first  production  items  less  than  during  EMD 
Respondent  69: 

a.  Only  preliminary  operators  manuals  were  required  for  bid  sample  testing.  Operator  through 
DS/GS  manuals  are  being  obtained  as  CDRL  requirements  under  the  production  contract 

Other: 

Respondent  10:  Use  capstone  Program  Management  Documentation  such  as  TEMPS,  ASRs,  and  ORDs 
a.  Allows  development  or  several  related  efforts  without  waiting  for  separate  staffing  and 
creation  of  multiple  documents 

Respondent  10:  Use  system  integration  of  existing  DoD  and  commercial  items  vs.  new  development 
a.  More  bang  for  the  buck,  quicker  time  to  field,  meets  the  intent  of  getting  equipment  into 
soldiers’  hands 

Respondent  22:  One  big  checklist  of  design  requirements 

a.  Too  many  separate  regulations  with  too  much  room  for  interpretation,  should  be  able  to  go 
down  list  and  address  each  requirement 

Respondent  69:  Cost  Savings 
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a.  Production  unit  costs  were  considerably  under  the  independent  Government  cost  estimate. 
The  incentive  of  a  multi-year  contract  and  the  inherent  competition  from  the  bid  sample  testing 
resulted  in  savings  of  approximately  40%  of  the  IGCE 
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No.  Copies 


1.  Defense  Technical  Information  Center  2 

Cameron  Station 

Alexandria,  VA  22304-6145 

2.  Library,  Code  52  2 

Naval  Postgraduate  School 

Monterey,  CA  93943-5101 

3.  Defense  Logistics  Studies  Information  Exchange  1 

U.S.  Army  Logistics  Management  College 

Ft.  Lee,  VA  23801-60433. 

4.  Dr.  David  V.  Lamm,  Code  SM/LT  5 

Naval  Postgraduate  School 

Monterey,  CA  93943 

5.  LTC  George  Prueitt,  Code  OR/PT  1 

Naval  Postgraduate  School 

Monterey,  CA  93943 

6.  Professor  Mark  Stone,  Code  SM/ST  2 

Naval  Postgraduate  School 

Monterey,  CA  93943 

7.  LTC  John  T.  Dillard,  Code  SM/DJ  1 

Naval  Postgraduate  School 

Monterey,  CA  93943 

8.  OASA  (RDA)  1 

ATTN:  SARD-ZAC 

103  Army  Pentagon 
Washington,  D.C.  20310 


9.  U.S.  Army  Natick  Research,  Development  and  Engineering  Center  3 

ATTN:  SATNC-AEL  (Ken  Rice) 

Natick,  MA  01760 
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10.  U.S.  Army  Natick  Research,  Development  and  Engineering  Center  1 

ATTN:  SATNC-UM  (Ed  Doucette) 

Natick,  MA  01760 

11.  U.S.  Army  Edgewood  Research,  Development  and  Engineering  Center  1 
ATTN:  SCBRD-ASA  (Dave  LaBar) 

Aberdeen  Proving  Ground,  MD  21010 

12.  U.S.  Army  Armaments  Research,  Development  and  Engineering  Center  1 
ATTN:  AMSTA-AR-RME-A  (Bill  Blackwell) 

Picatinny  Arsenal,  NJ  07806-5000 

13.  U.S.  Army  Tank,  Automotive  and  Armaments  RDEC  1 


10115  Gridley  Road,  Suite  128 
ATTN:  AMSTA-RB  (Jack  Petersen) 

Ft.  Belvoir,  VA  22060-5818 

14.  U.S.  Army  Aviation  Applied  Technology  Directorate  1 

ATTN:  AMSAT-R-TL  (Gene  Birocco) 

Ft.  Eustis,  VA  23604-5577 

15.  U.S.  Army  Communications  Electronics  Command  1 

ATTN:  AMSEL-PE-PD  (Nancy  Roberts) 

Ft.  Monmouth,  NJ  07703 

16.  U.S.  Army  Missile  Command  1 

ATTN:  AMSMI-WS  (Dr.  Gene  Paro) 

Redstone  Arsenal,  AL  35898-5798 
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